[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:31 dist-test-slave-rnrw krb5kdc[16554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:31 dist-test-slave-rnrw krb5kdc[16554](info): set up 2 sockets
Aug 12 18:53:31 dist-test-slave-rnrw krb5kdc[16554](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 18:53:32.808459 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:32 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:32.947839 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35877
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:33.101205 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:33.101591 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:33.101689 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:33.106479 16569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:33.106600 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:33.106637 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:33.106675 16569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:33.106735 16569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:33.113349 16569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:33.115065 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:33.116468 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:53:33.131968 16569 server_base.cc:1034] running on GCE node
W20260812 18:53:33.132052 16579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.132244 16582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.134076 16580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:33.134905 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:33.136885 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:33.138100 16569 hybrid_clock.cc:648] HybridClock initialized: now 1786560813138089 us; error 51 us; skew 500 ppm
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560813, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:33.141925 16569 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:33.143272 16569 webserver.cc:460] Webserver started at http://127.16.40.254:40083/ using document root <none> and password file <none>
I20260812 18:53:33.144258 16569 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:33.144364 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:33.144863 16569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:33.148039 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "376705484de84541a77238e6cb04edee"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "67be0798d84ca26379f4c1d5ad73c69f"
server_key_iv: "daee20851a3dc41efbde8ca288adc7fa"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.149298 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "376705484de84541a77238e6cb04edee"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "67be0798d84ca26379f4c1d5ad73c69f"
server_key_iv: "daee20851a3dc41efbde8ca288adc7fa"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.154946 16569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 18:53:33.158862 16590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.160823 16569 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:53:33.161039 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "376705484de84541a77238e6cb04edee"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "67be0798d84ca26379f4c1d5ad73c69f"
server_key_iv: "daee20851a3dc41efbde8ca288adc7fa"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.161181 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:33.174324 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:33.178650 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:33.178977 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:33.196425 16569 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:35877
I20260812 18:53:33.196493 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:35877 every 8 connection(s)
I20260812 18:53:33.197999 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:33.203507 16656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:33.209044 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 16569
I20260812 18:53:33.209244 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:33.211382 16547 external_mini_cluster.cc:1428] Setting key 4d942db2f266884953deebff8759ecb5
I20260812 18:53:33.214834 16656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee: Bootstrap starting.
I20260812 18:53:33.220000 16656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:33.221861 16656 log.cc:824] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:33.225244 16656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee: No bootstrap required, opened a new log
I20260812 18:53:33.233305 16656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } }
I20260812 18:53:33.233675 16656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:33.233844 16656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376705484de84541a77238e6cb04edee, State: Initialized, Role: FOLLOWER
I20260812 18:53:33.234673 16656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } }
I20260812 18:53:33.234913 16656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:33.235040 16656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:33.235183 16656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:33.236815 16656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } }
I20260812 18:53:33.237486 16656 leader_election.cc:302] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376705484de84541a77238e6cb04edee; no voters:
I20260812 18:53:33.237993 16656 leader_election.cc:288] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:33.238296 16661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:33.238796 16661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [term 1 LEADER]: Becoming Leader. State: Replica: 376705484de84541a77238e6cb04edee, State: Running, Role: LEADER
I20260812 18:53:33.239475 16661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } }
I20260812 18:53:33.239995 16656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:33.243011 16662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376705484de84541a77238e6cb04edee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } } }
I20260812 18:53:33.243233 16662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:33.244036 16663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376705484de84541a77238e6cb04edee. Latest consensus state: current_term: 1 leader_uuid: "376705484de84541a77238e6cb04edee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376705484de84541a77238e6cb04edee" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35877 } } }
I20260812 18:53:33.244150 16663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:33.245105 16671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:33.250896 16671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:33.251837 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:33.219295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.16.40.254:35877)
0812 18:53:33.220098 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:33.220139 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:33.221019 (+ 880us) server_negotiation.cc:407] Connection header received
0812 18:53:33.226630 (+ 5611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:33.226654 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:33.227269 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:33.227731 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:33.230793 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.232880 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.234724 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.235590 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.235799 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:33.242844 (+ 7045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:33.242893 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:33.242923 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:33.242979 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:33.246005 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.246941 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.246949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.246963 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.247150 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.247690 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.247693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.247695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.248535 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:33.248899 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:33.249800 (+ 901us) server_negotiation.cc:299] Negotiation successful
0812 18:53:33.250186 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":253,"threads_started":1}
I20260812 18:53:33.262568 16671 catalog_manager.cc:1347] Generated new cluster ID: c2555c2c8c62482f870fb6cef68ad08a
I20260812 18:53:33.262697 16671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:33.271286 16671 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:33.272688 16671 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:33.283222 16671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 376705484de84541a77238e6cb04edee: Generated new TSK 0
I20260812 18:53:33.284261 16671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:33.323776 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:33.468216 16686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:33.468532 16686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:33.468655 16686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:33.473030 16686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:33.473104 16686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:33.473169 16686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:33.478875 16686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:33.480084 16686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:33.481156 16686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:33.492393 16691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.492511 16692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.492540 16695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:33.492913 16686 server_base.cc:1034] running on GCE node
I20260812 18:53:33.493345 16686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:33.494076 16686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:33.495265 16686 hybrid_clock.cc:648] HybridClock initialized: now 1786560813495244 us; error 32 us; skew 500 ppm
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560813, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:33.498572 16686 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:33.499508 16686 webserver.cc:460] Webserver started at http://127.16.40.193:41399/ using document root <none> and password file <none>
I20260812 18:53:33.500180 16686 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:33.500279 16686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:33.500581 16686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:33.502777 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "4863c115a2f448c2b398de2226c289b7"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "efc5422e9cfc4459a0b4501a67461ad0"
server_key_iv: "7d954f8c91f678ff41536d1e8566dfc5"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.503588 16686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "4863c115a2f448c2b398de2226c289b7"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "efc5422e9cfc4459a0b4501a67461ad0"
server_key_iv: "7d954f8c91f678ff41536d1e8566dfc5"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.507863 16686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 18:53:33.510653 16705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.512073 16686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:33.512247 16686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "4863c115a2f448c2b398de2226c289b7"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "efc5422e9cfc4459a0b4501a67461ad0"
server_key_iv: "7d954f8c91f678ff41536d1e8566dfc5"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.512384 16686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:33.535185 16686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:33.539070 16686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:33.539324 16686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:33.540109 16686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:33.541518 16686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:33.541584 16686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.541663 16686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:33.541698 16686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.552735 16686 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:38151
I20260812 18:53:33.552795 16837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:38151 every 8 connection(s)
I20260812 18:53:33.553839 16686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:33.563637 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 16686
I20260812 18:53:33.563786 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:33.564088 16547 external_mini_cluster.cc:1428] Setting key c5ef6804b6d66e738a9e7a304d6c30fa
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560813, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:33.570381 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:33.556353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:41515 (local address 127.16.40.254:35877)
0812 18:53:33.556507 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:33.556511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:33.557517 (+ 1006us) server_negotiation.cc:407] Connection header received
0812 18:53:33.559134 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:33.559137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:33.559199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:33.559317 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:33.560566 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.561123 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.562300 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.562543 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.562681 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:33.565858 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:33.565862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:33.565865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:33.565897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:33.567585 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.568311 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.568315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.568317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.568419 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.568987 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.568990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.568992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.569214 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:33.569334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:33.570051 (+ 717us) server_negotiation.cc:299] Negotiation successful
0812 18:53:33.570201 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:53:33.571398 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:33.556851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35877 (local address 127.16.40.193:41515)
0812 18:53:33.557353 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:33.557399 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:33.558902 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:33.559503 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:33.559511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:33.559886 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:33.560388 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:33.560401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.561261 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.561265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:33.562163 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:33.562174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.562744 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.562748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:33.562926 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:33.563661 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:33.563678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:33.565686 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:33.567718 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:33.567730 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:33.567754 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:33.568176 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:33.568522 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:33.568527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:33.568529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:33.568827 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:33.569334 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:33.569343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:33.569749 (+ 406us) client_negotiation.cc:769] Sending connection context
0812 18:53:33.569972 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 18:53:33.570355 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
I20260812 18:53:33.573307 16838 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35877
I20260812 18:53:33.573577 16838 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:33.574160 16838 heartbeater.cc:511] Master 127.16.40.254:35877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:33.576958 16609 ts_manager.cc:194] Registered new tserver with Master: 4863c115a2f448c2b398de2226c289b7 (127.16.40.193:38151)
I20260812 18:53:33.578611 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:41515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:33.597436 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:33.735250 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:33.735595 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:33.735685 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:33.740381 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:33.740491 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:33.740633 16843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:33.746899 16843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:33.748093 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:33.749224 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:33.761082 16848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.761010 16852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:33.761046 16849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:33.761346 16843 server_base.cc:1034] running on GCE node
I20260812 18:53:33.761831 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:33.762562 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:33.763782 16843 hybrid_clock.cc:648] HybridClock initialized: now 1786560813763763 us; error 28 us; skew 500 ppm
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560813, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:33.767244 16843 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:33.768251 16843 webserver.cc:460] Webserver started at http://127.16.40.194:39993/ using document root <none> and password file <none>
I20260812 18:53:33.768929 16843 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:33.769025 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:33.769272 16843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:33.771782 16843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "2e4954a965324a39909c1d70ef0b799f"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "5b2e296f9d3ca2de032a9ad0848df0db"
server_key_iv: "0b377fe611418c02f7634db6542115b8"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.772468 16843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "2e4954a965324a39909c1d70ef0b799f"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "5b2e296f9d3ca2de032a9ad0848df0db"
server_key_iv: "0b377fe611418c02f7634db6542115b8"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.777139 16843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:53:33.780134 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.781620 16843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:33.781777 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "2e4954a965324a39909c1d70ef0b799f"
format_stamp: "Formatted at 2026-08-12 18:53:33 on dist-test-slave-rnrw"
server_key: "5b2e296f9d3ca2de032a9ad0848df0db"
server_key_iv: "0b377fe611418c02f7634db6542115b8"
server_key_version: "encryptionkey@0"
I20260812 18:53:33.781911 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:33.802865 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:33.806427 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:33.806651 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:33.807310 16843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:33.808265 16843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:33.808341 16843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.808424 16843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:33.808468 16843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:33.828852 16843 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:39867
I20260812 18:53:33.828975 16992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:39867 every 8 connection(s)
I20260812 18:53:33.829970 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:33.836156 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 16843
I20260812 18:53:33.836272 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:33.836596 16547 external_mini_cluster.cc:1428] Setting key 71040345b71688f42900b0faaea7daf1
Aug 12 18:53:33 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560813, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:33.848483 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:33.832094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:60421 (local address 127.16.40.254:35877)
0812 18:53:33.832250 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:33.832254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:33.833396 (+ 1142us) server_negotiation.cc:407] Connection header received
0812 18:53:33.834979 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:33.834982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:33.835036 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:33.835119 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:33.836819 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.837579 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.839430 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.839701 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.839798 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:33.843222 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:33.843230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:33.843236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:33.843272 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:33.845615 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.846221 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.846227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.846231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.846331 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:33.846966 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:33.846972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:33.846974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:33.847265 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:33.847408 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:33.848094 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 18:53:33.848257 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 18:53:33.849318 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:33.832510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35877 (local address 127.16.40.194:60421)
0812 18:53:33.833228 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:33.833275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:33.834761 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:33.835291 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:33.835297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:33.835816 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:33.836655 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:33.836678 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.837714 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.837717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:33.839293 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:33.839309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:33.839867 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:33.839874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:33.840112 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:33.841234 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:33.841250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:33.843027 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:33.845769 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:33.845775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:33.845790 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:33.846084 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:33.846632 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:33.846635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:33.846636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:33.846793 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:33.847458 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:33.847465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:33.847756 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 18:53:33.847932 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 18:53:33.848247 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":205,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:33.850800 16993 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35877
I20260812 18:53:33.851107 16993 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:33.851694 16993 heartbeater.cc:511] Master 127.16.40.254:35877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:53:33.852965 16609 ts_manager.cc:194] Registered new tserver with Master: 2e4954a965324a39909c1d70ef0b799f (127.16.40.194:39867)
I20260812 18:53:33.853839 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:60421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:33.872325 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:33.994637 16998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:33.994957 16998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:33.995043 16998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:33.999291 16998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:33.999378 16998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:33.999480 16998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:34.005156 16998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35877
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:34.006403 16998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:34.007556 16998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:53:34.018206 16998 server_base.cc:1034] running on GCE node
W20260812 18:53:34.018313 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:34.018201 17006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:34.018467 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:34.018908 16998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:34.019650 16998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:34.020823 16998 hybrid_clock.cc:648] HybridClock initialized: now 1786560814020783 us; error 53 us; skew 500 ppm
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:34.024161 16998 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:34.025416 16998 webserver.cc:460] Webserver started at http://127.16.40.195:34397/ using document root <none> and password file <none>
I20260812 18:53:34.026026 16998 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:34.026131 16998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:34.026377 16998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:34.028697 16998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "f9e3616dc7bb4e8b9408c0ef4ada8e2c"
format_stamp: "Formatted at 2026-08-12 18:53:34 on dist-test-slave-rnrw"
server_key: "8aec9c6d5cb0791b2d620cb6d6bb744b"
server_key_iv: "01d1f519b0877425ab154a7a593b8c5d"
server_key_version: "encryptionkey@0"
I20260812 18:53:34.029297 16998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "f9e3616dc7bb4e8b9408c0ef4ada8e2c"
format_stamp: "Formatted at 2026-08-12 18:53:34 on dist-test-slave-rnrw"
server_key: "8aec9c6d5cb0791b2d620cb6d6bb744b"
server_key_iv: "01d1f519b0877425ab154a7a593b8c5d"
server_key_version: "encryptionkey@0"
I20260812 18:53:34.033401 16998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:34.035982 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:34.037324 16998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 18:53:34.037470 16998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "f9e3616dc7bb4e8b9408c0ef4ada8e2c"
format_stamp: "Formatted at 2026-08-12 18:53:34 on dist-test-slave-rnrw"
server_key: "8aec9c6d5cb0791b2d620cb6d6bb744b"
server_key_iv: "01d1f519b0877425ab154a7a593b8c5d"
server_key_version: "encryptionkey@0"
I20260812 18:53:34.037591 16998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:34.048453 16998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:34.051987 16998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:34.052222 16998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:34.052954 16998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:34.053865 16998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:34.053939 16998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:34.054013 16998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:34.054085 16998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:34.063913 16998 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:37047
I20260812 18:53:34.063952 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:37047 every 8 connection(s)
I20260812 18:53:34.064966 16998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:34.070187 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 16998
I20260812 18:53:34.070333 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:34.070716 16547 external_mini_cluster.cc:1428] Setting key a0c6b647769a53310748269cfc915e61
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:34.079773 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:34.067026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43077 (local address 127.16.40.254:35877)
0812 18:53:34.067208 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:34.067214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:34.068727 (+ 1513us) server_negotiation.cc:407] Connection header received
0812 18:53:34.069587 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:34.069590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:34.069640 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:34.069720 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:34.071474 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.072148 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.073410 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.073704 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.073820 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:34.076414 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:34.076418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:34.076420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:34.076445 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:34.077898 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.078394 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.078396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.078397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.078475 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.078790 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.078792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.078793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.078971 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:34.079048 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:34.079529 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 18:53:34.079643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 18:53:34.080760 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:34.067421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35877 (local address 127.16.40.195:43077)
0812 18:53:34.068501 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:34.068597 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:34.069380 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:34.069957 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:34.069969 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:34.070530 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:34.071291 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:34.071311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.072285 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.072289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:34.073259 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:34.073293 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.073860 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.073863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:34.074017 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:34.074596 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:34.074613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:34.076243 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:34.078006 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:34.078012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:34.078025 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:34.078303 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:34.078558 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:34.078560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:34.078561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:34.078706 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:34.079059 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:34.079064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:34.079309 (+ 245us) client_negotiation.cc:769] Sending connection context
0812 18:53:34.079470 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 18:53:34.079734 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":724,"threads_started":1}
I20260812 18:53:34.081895 17158 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35877
I20260812 18:53:34.082167 17158 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:34.082700 17158 heartbeater.cc:511] Master 127.16.40.254:35877 requested a full tablet report, sending...
I20260812 18:53:34.083794 16609 ts_manager.cc:194] Registered new tserver with Master: f9e3616dc7bb4e8b9408c0ef4ada8e2c (127.16.40.195:37047)
I20260812 18:53:34.084625 16609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:43077
I20260812 18:53:34.086083 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:34.117558 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:34.105834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.16.40.193:38151)
0812 18:53:34.106196 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:34.106203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:34.106229 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:53:34.106282 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:34.106288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:34.106492 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:34.106690 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:34.107553 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.108415 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.109418 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.109916 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.110027 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:34.113104 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:34.113118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:34.113136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:34.113173 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:34.115193 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.115823 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.115830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.115832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.115934 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.116300 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.116304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.116306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.116697 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:34.116946 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:34.117156 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 18:53:34.117339 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
W20260812 18:53:34.118886 16737 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43156
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[16554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:34.148906 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:34.139214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.16.40.193:38151)
0812 18:53:34.139372 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:34.139376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:34.139591 (+ 215us) server_negotiation.cc:407] Connection header received
0812 18:53:34.139649 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:34.139651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:34.139700 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:34.139784 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:34.140667 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.141255 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.142200 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:34.142490 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:34.142580 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:34.145590 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:34.145596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:34.145599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:34.145636 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:34.146939 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.147375 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.147377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.147378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.147469 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:34.147752 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:34.147755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:34.147756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:34.147953 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:34.148074 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:34.148484 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 18:53:34.148681 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:53:34.150202 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 16686
I20260812 18:53:34.159829 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 16843
I20260812 18:53:34.169320 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 16998
I20260812 18:53:34.181713 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 16569
2026-08-12T18:53:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2609 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[17177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[17177](info): set up 2 sockets
Aug 12 18:53:34 dist-test-slave-rnrw krb5kdc[17177](info): commencing operation
krb5kdc: starting...
W20260812 18:53:35.318673 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:35 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:35.460311 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41781
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:35.586790 17190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:35.587167 17190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:35.587261 17190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:35.591624 17190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:35.591741 17190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:35.591779 17190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:35.591822 17190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:35.591859 17190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:35.597581 17190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:35.598812 17190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:35.599917 17190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:35.608911 17197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:35.608978 17198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:35.608933 17200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:35.609292 17190 server_base.cc:1034] running on GCE node
I20260812 18:53:35.610061 17190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:35.611285 17190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:35.612493 17190 hybrid_clock.cc:648] HybridClock initialized: now 1786560815612478 us; error 39 us; skew 500 ppm
Aug 12 18:53:35 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:35.620173 17190 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:35.621358 17190 webserver.cc:460] Webserver started at http://127.16.40.254:42611/ using document root <none> and password file <none>
I20260812 18:53:35.621964 17190 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:35.622036 17190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:35.622220 17190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:35.624347 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "ecd87b3e651545d5800fb525ee0c441c"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "4f347d6e424bfa5615e850402a8190a2"
server_key_iv: "d55586c86fa4a7c423ab0777d5614de1"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.624974 17190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "ecd87b3e651545d5800fb525ee0c441c"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "4f347d6e424bfa5615e850402a8190a2"
server_key_iv: "d55586c86fa4a7c423ab0777d5614de1"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.629076 17190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:53:35.631804 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:35.633556 17190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:35.633724 17190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "ecd87b3e651545d5800fb525ee0c441c"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "4f347d6e424bfa5615e850402a8190a2"
server_key_iv: "d55586c86fa4a7c423ab0777d5614de1"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.633867 17190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:35.668512 17190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:35.672214 17190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:35.672446 17190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:35.680266 17190 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:41781
I20260812 18:53:35.680295 17271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:41781 every 8 connection(s)
I20260812 18:53:35.681375 17190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:35.684407 17272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:35.689582 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17190
I20260812 18:53:35.689684 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:35.689951 16547 external_mini_cluster.cc:1428] Setting key 651e57446861d07c3fc27a6a00abba88
I20260812 18:53:35.690604 17272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: Bootstrap starting.
I20260812 18:53:35.693487 17272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:35.694471 17272 log.cc:824] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:35.696976 17272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: No bootstrap required, opened a new log
Aug 12 18:53:35 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:35.701027 17272 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } }
I20260812 18:53:35.701402 17272 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:35.701498 17272 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd87b3e651545d5800fb525ee0c441c, State: Initialized, Role: FOLLOWER
I20260812 18:53:35.702080 17272 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } }
I20260812 18:53:35.702247 17272 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:35.702340 17272 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:35.702451 17272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:35.703500 17272 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } }
I20260812 18:53:35.703896 17272 leader_election.cc:302] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd87b3e651545d5800fb525ee0c441c; no voters:
I20260812 18:53:35.704308 17272 leader_election.cc:288] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:35.704427 17278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:35.704721 17278 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [term 1 LEADER]: Becoming Leader. State: Replica: ecd87b3e651545d5800fb525ee0c441c, State: Running, Role: LEADER
I20260812 18:53:35.705049 17278 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } }
I20260812 18:53:35.705857 17272 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:35.706352 17278 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd87b3e651545d5800fb525ee0c441c. Latest consensus state: current_term: 1 leader_uuid: "ecd87b3e651545d5800fb525ee0c441c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } } }
I20260812 18:53:35.706482 17278 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:35.707086 17281 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd87b3e651545d5800fb525ee0c441c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd87b3e651545d5800fb525ee0c441c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41781 } } }
I20260812 18:53:35.707258 17281 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:35.707444 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:35.691564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.16.40.254:41781)
0812 18:53:35.692074 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:35.692096 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:35.692142 (+ 46us) server_negotiation.cc:407] Connection header received
0812 18:53:35.693042 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:35.693097 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:35.693549 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:35.693950 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:35.694805 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.696187 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.697213 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.697752 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.697952 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:35.700642 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:35.700649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:35.700667 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:35.700700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:35.703026 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:35.703566 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:35.703573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:35.703580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:35.703769 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:35.704130 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:35.704134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:35.704140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:35.704793 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:35.705690 (+ 897us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:35.705851 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 18:53:35.706229 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":144,"threads_started":1}
W20260812 18:53:35.711308 17297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:53:35.711414 17297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:53:35.711491 17284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:35.712353 17284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:35.718672 17284 catalog_manager.cc:1347] Generated new cluster ID: 1ca66838ad29483699498d19961e7a48
I20260812 18:53:35.718766 17284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:35.727087 17284 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:35.728152 17284 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:35.738847 17284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd87b3e651545d5800fb525ee0c441c: Generated new TSK 0
I20260812 18:53:35.739840 17284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:35.779780 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:35.900974 17303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:35.901312 17303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:35.901360 17303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:35.905612 17303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:35.905699 17303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:35.905763 17303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:35.911790 17303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:35.913038 17303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:35.914148 17303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:35.924358 17311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:35.924634 17314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:35.924397 17303 server_base.cc:1034] running on GCE node
W20260812 18:53:35.924357 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:35.925182 17303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:35.925905 17303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:35.927110 17303 hybrid_clock.cc:648] HybridClock initialized: now 1786560815927070 us; error 51 us; skew 500 ppm
Aug 12 18:53:35 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:35.930150 17303 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:35.931103 17303 webserver.cc:460] Webserver started at http://127.16.40.193:46771/ using document root <none> and password file <none>
I20260812 18:53:35.931706 17303 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:35.931802 17303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:35.932049 17303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:35.934218 17303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "25a8e09539444187864d933eadc0e051"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "a4ec7d0702f43405fa4a0e643b29ebcb"
server_key_iv: "a3c5400a900b751395dc644a7f87470e"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.934819 17303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "25a8e09539444187864d933eadc0e051"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "a4ec7d0702f43405fa4a0e643b29ebcb"
server_key_iv: "a3c5400a900b751395dc644a7f87470e"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.938885 17303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 18:53:35.941622 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:35.942979 17303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:35.943130 17303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "25a8e09539444187864d933eadc0e051"
format_stamp: "Formatted at 2026-08-12 18:53:35 on dist-test-slave-rnrw"
server_key: "a4ec7d0702f43405fa4a0e643b29ebcb"
server_key_iv: "a3c5400a900b751395dc644a7f87470e"
server_key_version: "encryptionkey@0"
I20260812 18:53:35.943255 17303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:35.964398 17303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:35.968355 17303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:35.968698 17303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:35.969430 17303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:35.970422 17303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:35.970499 17303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:35.970602 17303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:35.970654 17303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:35.980604 17303 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:36949
I20260812 18:53:35.980626 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:36949 every 8 connection(s)
I20260812 18:53:35.981632 17303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:35.987973 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17303
I20260812 18:53:35.988116 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:35.988437 16547 external_mini_cluster.cc:1428] Setting key 8ec6572d28de1e2fd060244e1103c1e1
Aug 12 18:53:35 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:35.999800 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:35.983800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:33985 (local address 127.16.40.254:41781)
0812 18:53:35.984000 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:35.984005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:35.985055 (+ 1050us) server_negotiation.cc:407] Connection header received
0812 18:53:35.986463 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:35.986468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:35.986525 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:35.986646 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:35.988650 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.989243 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.990899 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.991378 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.991509 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:35.995037 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:35.995041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:35.995043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:35.995073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:35.996820 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:35.997664 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:35.997668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:35.997669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:35.997762 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:35.998323 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:35.998327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:35.998328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:35.998547 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:35.998661 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:35.999507 (+ 846us) server_negotiation.cc:299] Negotiation successful
0812 18:53:35.999645 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:36.001190 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:35.984294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41781 (local address 127.16.40.193:33985)
0812 18:53:35.984867 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:35.984915 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:35.986169 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:35.986965 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:35.986974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:35.987542 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:35.988379 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:35.988405 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.989495 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.989499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:35.990744 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:35.990755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:35.991607 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:35.991610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:35.992170 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:35.992962 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:35.992981 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:35.994763 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:35.996985 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:35.996996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:35.997023 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:35.997513 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:35.997904 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:35.997911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:35.997915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:35.998169 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:35.998695 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:35.998706 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:35.999152 (+ 446us) client_negotiation.cc:769] Sending connection context
0812 18:53:35.999431 (+ 279us) client_negotiation.cc:240] Negotiation successful
0812 18:53:35.999893 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":205,"threads_started":1}
I20260812 18:53:36.002921 17460 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41781
I20260812 18:53:36.003178 17460 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:36.003863 17460 heartbeater.cc:511] Master 127.16.40.254:41781 requested a full tablet report, sending...
I20260812 18:53:36.005954 17227 ts_manager.cc:194] Registered new tserver with Master: 25a8e09539444187864d933eadc0e051 (127.16.40.193:36949)
I20260812 18:53:36.007814 17227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:33985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
W20260812 18:53:36.765803 16547 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.40.194: real 0.775s user 0.005s sys 0.000s
I20260812 18:53:36.766381 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:36.891485 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:36.891803 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:36.891892 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:36.896262 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:36.896345 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:36.896459 17467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:36.902088 17467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:36.903342 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:36.904479 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:36.915200 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:36.915189 17476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:36.915235 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:36.915442 17467 server_base.cc:1034] running on GCE node
I20260812 18:53:36.915956 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:36.916800 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:36.917979 17467 hybrid_clock.cc:648] HybridClock initialized: now 1786560816917957 us; error 31 us; skew 500 ppm
Aug 12 18:53:36 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:36.921576 17467 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:36.922554 17467 webserver.cc:460] Webserver started at http://127.16.40.194:45505/ using document root <none> and password file <none>
I20260812 18:53:36.923128 17467 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:36.923193 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:36.923369 17467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:36.925645 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "82489a49a341410fa065ef53a6e4cfa2"
format_stamp: "Formatted at 2026-08-12 18:53:36 on dist-test-slave-rnrw"
server_key: "2edf6db93b63ecfead6dad033f48e680"
server_key_iv: "7a7a9d189124a019b050b61baf38cbd8"
server_key_version: "encryptionkey@0"
I20260812 18:53:36.926190 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "82489a49a341410fa065ef53a6e4cfa2"
format_stamp: "Formatted at 2026-08-12 18:53:36 on dist-test-slave-rnrw"
server_key: "2edf6db93b63ecfead6dad033f48e680"
server_key_iv: "7a7a9d189124a019b050b61baf38cbd8"
server_key_version: "encryptionkey@0"
I20260812 18:53:36.930703 17467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:36.933605 17486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:36.934995 17467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 18:53:36.935155 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "82489a49a341410fa065ef53a6e4cfa2"
format_stamp: "Formatted at 2026-08-12 18:53:36 on dist-test-slave-rnrw"
server_key: "2edf6db93b63ecfead6dad033f48e680"
server_key_iv: "7a7a9d189124a019b050b61baf38cbd8"
server_key_version: "encryptionkey@0"
I20260812 18:53:36.935686 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:36.965051 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:36.968394 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:36.968696 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:36.969422 17467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:36.970427 17467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:36.970508 17467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:36.970602 17467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:36.970655 17467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:36.981935 17467 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:39191
I20260812 18:53:36.981964 17616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:39191 every 8 connection(s)
I20260812 18:53:36.983040 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:36.984918 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17467
I20260812 18:53:36.985069 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:36.985401 16547 external_mini_cluster.cc:1428] Setting key 04f547931149c6d4874787291562ccaa
Aug 12 18:53:36 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560816, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:36.999132 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:36.985269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:59647 (local address 127.16.40.254:41781)
0812 18:53:36.985542 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:36.985547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:36.986689 (+ 1142us) server_negotiation.cc:407] Connection header received
0812 18:53:36.988140 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:36.988143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:36.988201 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:36.988292 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:36.989582 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:36.990134 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:36.991433 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:36.991692 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:36.991836 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:36.994969 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:36.994975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:36.994985 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:36.995014 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:36.996898 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:36.997525 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:36.997527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:36.997529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:36.997619 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:36.997944 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:36.997947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:36.997948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:36.998188 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:36.998276 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:36.998828 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 18:53:36.998955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
I20260812 18:53:36.999964 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:36.985862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41781 (local address 127.16.40.194:59647)
0812 18:53:36.986535 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:36.986588 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:36.987892 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:36.988470 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:36.988478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:36.988918 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:36.989422 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:36.989435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:36.990271 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:36.990280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:36.991286 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:36.991300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:36.991875 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:36.991882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:36.992130 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:36.992879 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:36.992906 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:36.994754 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:36.997067 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:36.997077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:36.997099 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:36.997422 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:36.997708 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:36.997710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:36.997712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:36.997855 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:36.998285 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:36.998291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:36.998559 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 18:53:36.998730 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 18:53:36.999021 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":214,"threads_started":1}
I20260812 18:53:37.001503 17617 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41781
I20260812 18:53:37.001783 17617 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:37.002336 17617 heartbeater.cc:511] Master 127.16.40.254:41781 requested a full tablet report, sending...
I20260812 18:53:37.003595 17227 ts_manager.cc:194] Registered new tserver with Master: 82489a49a341410fa065ef53a6e4cfa2 (127.16.40.194:39191)
I20260812 18:53:37.004240 17227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:59647
I20260812 18:53:37.011122 17460 heartbeater.cc:503] Master 127.16.40.254:41781 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
W20260812 18:53:37.770970 16547 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.40.195: real 0.783s user 0.000s sys 0.004s
I20260812 18:53:37.771569 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:37.891326 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:37.891623 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:37.891669 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:37.895799 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:37.895871 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:37.895933 17625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:37.901328 17625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41781
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:37.902405 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:37.903419 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:37.914266 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:37.914266 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:37.914271 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:37.914328 17625 server_base.cc:1034] running on GCE node
I20260812 18:53:37.915048 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:37.915752 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:37.916975 17625 hybrid_clock.cc:648] HybridClock initialized: now 1786560817916925 us; error 59 us; skew 500 ppm
Aug 12 18:53:37 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:37.919860 17625 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:37.920887 17625 webserver.cc:460] Webserver started at http://127.16.40.195:33557/ using document root <none> and password file <none>
I20260812 18:53:37.921525 17625 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:37.921626 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:37.921883 17625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:37.924012 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "8c4a050503d24f4889bb25f56c9ddb34"
format_stamp: "Formatted at 2026-08-12 18:53:37 on dist-test-slave-rnrw"
server_key: "a2dd4beb19a9a705a41722aced867611"
server_key_iv: "4ec46f63fa8221617d34dc1306982706"
server_key_version: "encryptionkey@0"
I20260812 18:53:37.924682 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "8c4a050503d24f4889bb25f56c9ddb34"
format_stamp: "Formatted at 2026-08-12 18:53:37 on dist-test-slave-rnrw"
server_key: "a2dd4beb19a9a705a41722aced867611"
server_key_iv: "4ec46f63fa8221617d34dc1306982706"
server_key_version: "encryptionkey@0"
I20260812 18:53:37.928659 17625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:53:37.931252 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:37.932536 17625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 18:53:37.932753 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "8c4a050503d24f4889bb25f56c9ddb34"
format_stamp: "Formatted at 2026-08-12 18:53:37 on dist-test-slave-rnrw"
server_key: "a2dd4beb19a9a705a41722aced867611"
server_key_iv: "4ec46f63fa8221617d34dc1306982706"
server_key_version: "encryptionkey@0"
I20260812 18:53:37.932873 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:37.953703 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:37.957414 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:37.957721 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:37.958400 17625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:37.959323 17625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:37.959398 17625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:37.959473 17625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:37.959527 17625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:37.969237 17625 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:40551
I20260812 18:53:37.969277 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:40551 every 8 connection(s)
I20260812 18:53:37.970348 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:37.979722 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17625
I20260812 18:53:37.979890 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:37.980286 16547 external_mini_cluster.cc:1428] Setting key 88f761c133838d2f8e3d0886c7ac5c3b
Aug 12 18:53:37 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560817, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:37.985795 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:37.972447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:40355 (local address 127.16.40.254:41781)
0812 18:53:37.972804 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:37.972808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:37.973520 (+ 712us) server_negotiation.cc:407] Connection header received
0812 18:53:37.974955 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:37.974958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:37.975022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:37.975137 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:37.976363 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:37.976979 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:37.978187 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:37.978429 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:37.978556 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:37.981468 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:37.981472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:37.981475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:37.981507 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:37.983398 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:37.983972 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:37.983980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:37.983982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:37.984076 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:37.984582 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:37.984586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:37.984588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:37.984807 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:37.984920 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:37.985447 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 18:53:37.985593 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260812 18:53:37.986634 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:37.972890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41781 (local address 127.16.40.195:40355)
0812 18:53:37.973366 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:37.973412 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:37.974724 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:37.975321 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:37.975329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:37.975714 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:37.976214 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:37.976228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:37.977111 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:37.977115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:37.978058 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:37.978068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:37.978596 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:37.978600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:37.978776 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:37.979370 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:37.979387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:37.981269 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:37.983545 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:37.983552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:37.983569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:37.983854 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:37.984279 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:37.984282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:37.984284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:37.984448 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:37.984921 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:37.984929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:37.985201 (+ 272us) client_negotiation.cc:769] Sending connection context
0812 18:53:37.985401 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 18:53:37.985768 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260812 18:53:37.987809 17775 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41781
I20260812 18:53:37.988062 17775 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:37.988694 17775 heartbeater.cc:511] Master 127.16.40.254:41781 requested a full tablet report, sending...
I20260812 18:53:37.989801 17227 ts_manager.cc:194] Registered new tserver with Master: 8c4a050503d24f4889bb25f56c9ddb34 (127.16.40.195:40551)
I20260812 18:53:37.990455 17227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:40355
I20260812 18:53:37.994836 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:53:38.006510 17617 heartbeater.cc:503] Master 127.16.40.254:41781 was elected leader, sending a full tablet report...
I20260812 18:53:38.007967 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:37.999586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.16.40.254:41781)
0812 18:53:37.999773 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:37.999779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:37.999936 (+ 157us) server_negotiation.cc:407] Connection header received
0812 18:53:38.000119 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.000127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.000173 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.000267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:38.001131 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.001672 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.002554 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.002790 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.002904 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.004344 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:38.004349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:38.004357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:38.004382 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:38.005832 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.006314 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.006322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.006324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.006437 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.006842 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.006848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.006850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.007203 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:38.007351 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.007618 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.007776 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 18:53:38.014346 17227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:38.017925 17227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:38.035153 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.030340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.16.40.194:39191)
0812 18:53:38.030731 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.030736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.030758 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:53:38.030834 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.030839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.031000 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.031173 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:38.031532 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032410 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.033858 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.034438 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.034580 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.034836 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.034895 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.034989 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":133,"threads_started":1}
I20260812 18:53:38.035145 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.029901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39191 (local address 127.0.0.1:45784)
0812 18:53:38.030486 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:38.030498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:38.030581 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:38.031316 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:38.031321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:38.031343 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:38.031407 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.031411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032565 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.032569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.033721 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.033732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.034581 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.034585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.034693 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.034759 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 18:53:38.034869 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 18:53:38.035003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":78,"thread_start_us":96,"threads_started":1}
I20260812 18:53:38.035717 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.029083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:40551 (local address 127.0.0.1:49904)
0812 18:53:38.029992 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:38.030049 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:38.030185 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:38.030988 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:38.030991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:38.031004 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:38.031191 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.031201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032683 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.032686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.034174 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.034186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.035316 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.035320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.035408 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.035431 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:53:38.035491 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 18:53:38.035564 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":132,"thread_start_us":127,"threads_started":1}
I20260812 18:53:38.035895 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.029494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:36949 (local address 127.0.0.1:45968)
0812 18:53:38.030222 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:38.030236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:38.030311 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:38.030890 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:38.030900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:38.030929 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:38.031147 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.031156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032682 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.032686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.034173 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.034186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.035539 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.035542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.035621 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.035643 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 18:53:38.035691 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 18:53:38.035761 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":169,"thread_start_us":140,"threads_started":1}
I20260812 18:53:38.036590 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.030001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.16.40.193:36949)
0812 18:53:38.030369 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.030379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.030405 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:53:38.030489 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.030495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.030718 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.030883 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:38.031292 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032506 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.034322 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.035168 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.036011 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.036248 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.036311 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.036403 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":63,"threads_started":1}
I20260812 18:53:38.037041 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.029277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.16.40.195:40551)
0812 18:53:38.030523 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.030538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.030564 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:53:38.030645 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.030652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.030838 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.031113 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:38.031280 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.032506 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.034322 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.035168 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.036559 (+ 1391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.036731 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.036803 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.036901 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":250,"threads_started":1}
I20260812 18:53:38.038916 17544 tablet_service.cc:1467] Processing CreateTablet for tablet b1636998a6624a2dbc01e6127d1877f3 (DEFAULT_TABLE table=test-table [id=673766a033f0463d8fbfa60a3d4da0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:38.039312 17380 tablet_service.cc:1467] Processing CreateTablet for tablet b1636998a6624a2dbc01e6127d1877f3 (DEFAULT_TABLE table=test-table [id=673766a033f0463d8fbfa60a3d4da0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:38.039654 17701 tablet_service.cc:1467] Processing CreateTablet for tablet b1636998a6624a2dbc01e6127d1877f3 (DEFAULT_TABLE table=test-table [id=673766a033f0463d8fbfa60a3d4da0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:38.040432 17544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1636998a6624a2dbc01e6127d1877f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:38.040453 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1636998a6624a2dbc01e6127d1877f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:38.040650 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1636998a6624a2dbc01e6127d1877f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:38.047349 17795 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: Bootstrap starting.
I20260812 18:53:38.048338 17796 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: Bootstrap starting.
I20260812 18:53:38.049758 17795 tablet_bootstrap.cc:654] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:38.050274 17796 tablet_bootstrap.cc:654] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:38.050746 17795 log.cc:824] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:38.051093 17796 log.cc:824] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:38.051613 17797 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: Bootstrap starting.
I20260812 18:53:38.053097 17796 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: No bootstrap required, opened a new log
I20260812 18:53:38.053324 17796 ts_tablet_manager.cc:1397] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 18:53:38.053829 17797 tablet_bootstrap.cc:654] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:38.053920 17795 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: No bootstrap required, opened a new log
I20260812 18:53:38.054126 17795 ts_tablet_manager.cc:1397] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 18:53:38.055375 17797 log.cc:824] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:38.057658 17796 raft_consensus.cc:348] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.057924 17797 tablet_bootstrap.cc:492] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: No bootstrap required, opened a new log
I20260812 18:53:38.057966 17796 raft_consensus.cc:374] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:38.058024 17796 raft_consensus.cc:729] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82489a49a341410fa065ef53a6e4cfa2, State: Initialized, Role: FOLLOWER
I20260812 18:53:38.058131 17797 ts_tablet_manager.cc:1397] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:53:38.058573 17796 consensus_queue.cc:260] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.058817 17795 raft_consensus.cc:348] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.059103 17795 raft_consensus.cc:374] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:38.059172 17795 raft_consensus.cc:729] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a8e09539444187864d933eadc0e051, State: Initialized, Role: FOLLOWER
I20260812 18:53:38.059697 17796 ts_tablet_manager.cc:1428] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:53:38.059698 17795 consensus_queue.cc:260] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.061756 17795 ts_tablet_manager.cc:1428] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 18:53:38.063179 17797 raft_consensus.cc:348] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.063464 17797 raft_consensus.cc:374] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:38.063542 17797 raft_consensus.cc:729] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4a050503d24f4889bb25f56c9ddb34, State: Initialized, Role: FOLLOWER
I20260812 18:53:38.064055 17797 consensus_queue.cc:260] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.064759 17775 heartbeater.cc:503] Master 127.16.40.254:41781 was elected leader, sending a full tablet report...
I20260812 18:53:38.065114 17797 ts_tablet_manager.cc:1428] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 18:53:38.081009 17807 raft_consensus.cc:482] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:38.081233 17807 raft_consensus.cc:504] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.082480 17807 leader_election.cc:288] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c4a050503d24f4889bb25f56c9ddb34 (127.16.40.195:40551), 82489a49a341410fa065ef53a6e4cfa2 (127.16.40.194:39191)
I20260812 18:53:38.087258 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.082836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:40551 (local address 127.16.40.193:35385)
0812 18:53:38.083208 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:38.083224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:38.083354 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:38.083752 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:38.083758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:38.083784 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:38.083965 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.083973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.085005 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.085009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.085972 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.085982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.086782 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.086786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.086887 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.086916 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 18:53:38.086997 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 18:53:38.087113 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":212,"threads_started":1}
I20260812 18:53:38.087297 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.082916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:35385 (local address 127.16.40.195:40551)
0812 18:53:38.083091 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.083095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.083310 (+ 215us) server_negotiation.cc:407] Connection header received
0812 18:53:38.083480 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.083486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.083617 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.083724 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:38.084106 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.084889 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.086109 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.086659 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.086758 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.086805 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.087062 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.087152 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:53:38.087587 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.083692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:33049 (local address 127.16.40.194:39191)
0812 18:53:38.083832 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.083838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.083854 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:53:38.084014 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.084019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.084077 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.084147 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:38.084496 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.085200 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.086423 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.086911 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.087032 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.087078 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.087336 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.087442 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:53:38.087885 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.083387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39191 (local address 127.16.40.193:33049)
0812 18:53:38.083778 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:38.083794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:38.083902 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:38.084201 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:38.084204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:38.084225 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:38.084364 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.084372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.085325 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.085328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.086287 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:38.086301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.087072 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.087078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:38.087186 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.087209 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:53:38.087299 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 18:53:38.087731 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":140,"threads_started":1}
I20260812 18:53:38.088234 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1636998a6624a2dbc01e6127d1877f3" candidate_uuid: "25a8e09539444187864d933eadc0e051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4a050503d24f4889bb25f56c9ddb34" is_pre_election: true
I20260812 18:53:38.088728 17727 raft_consensus.cc:2454] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25a8e09539444187864d933eadc0e051 in term 0.
I20260812 18:53:38.088716 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1636998a6624a2dbc01e6127d1877f3" candidate_uuid: "25a8e09539444187864d933eadc0e051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82489a49a341410fa065ef53a6e4cfa2" is_pre_election: true
I20260812 18:53:38.088989 17567 raft_consensus.cc:2454] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25a8e09539444187864d933eadc0e051 in term 0.
I20260812 18:53:38.089416 17328 leader_election.cc:302] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a8e09539444187864d933eadc0e051, 8c4a050503d24f4889bb25f56c9ddb34; no voters:
I20260812 18:53:38.089713 17807 raft_consensus.cc:2781] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:38.089823 17807 raft_consensus.cc:482] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:38.089880 17807 raft_consensus.cc:3037] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:38.091439 17807 raft_consensus.cc:504] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.091809 17807 leader_election.cc:288] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [CANDIDATE]: Term 1 election: Requested vote from peers 8c4a050503d24f4889bb25f56c9ddb34 (127.16.40.195:40551), 82489a49a341410fa065ef53a6e4cfa2 (127.16.40.194:39191)
I20260812 18:53:38.092412 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1636998a6624a2dbc01e6127d1877f3" candidate_uuid: "25a8e09539444187864d933eadc0e051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82489a49a341410fa065ef53a6e4cfa2"
I20260812 18:53:38.092427 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1636998a6624a2dbc01e6127d1877f3" candidate_uuid: "25a8e09539444187864d933eadc0e051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c4a050503d24f4889bb25f56c9ddb34"
I20260812 18:53:38.092586 17567 raft_consensus.cc:3037] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:38.092608 17727 raft_consensus.cc:3037] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:38.093721 17727 raft_consensus.cc:2454] T b1636998a6624a2dbc01e6127d1877f3 P 8c4a050503d24f4889bb25f56c9ddb34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25a8e09539444187864d933eadc0e051 in term 1.
I20260812 18:53:38.093753 17567 raft_consensus.cc:2454] T b1636998a6624a2dbc01e6127d1877f3 P 82489a49a341410fa065ef53a6e4cfa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25a8e09539444187864d933eadc0e051 in term 1.
I20260812 18:53:38.094095 17328 leader_election.cc:302] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a8e09539444187864d933eadc0e051, 82489a49a341410fa065ef53a6e4cfa2; no voters:
I20260812 18:53:38.094319 17807 raft_consensus.cc:2781] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:38.094594 17807 raft_consensus.cc:686] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [term 1 LEADER]: Becoming Leader. State: Replica: 25a8e09539444187864d933eadc0e051, State: Running, Role: LEADER
I20260812 18:53:38.094986 17807 consensus_queue.cc:237] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } }
I20260812 18:53:38.098860 17228 catalog_manager.cc:5495] T b1636998a6624a2dbc01e6127d1877f3 P 25a8e09539444187864d933eadc0e051 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25a8e09539444187864d933eadc0e051 (127.16.40.193). New cstate: current_term: 1 leader_uuid: "25a8e09539444187864d933eadc0e051" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c4a050503d24f4889bb25f56c9ddb34" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 40551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82489a49a341410fa065ef53a6e4cfa2" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25a8e09539444187864d933eadc0e051" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36949 } health_report { overall_health: HEALTHY } } }
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:38.119752 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.108367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.16.40.193:36949)
0812 18:53:38.108600 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.108606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.108701 (+ 95us) server_negotiation.cc:407] Connection header received
0812 18:53:38.108819 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.108822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.108867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.108939 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:38.109730 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.110257 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.111193 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.111730 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.111841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.114777 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:38.114795 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:38.114812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:38.114844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:38.117059 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.117735 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.117742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.117743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.117885 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.118418 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.118422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.118423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.118739 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:38.118953 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.119273 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.119537 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:38.152731 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.143742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.16.40.193:36949)
0812 18:53:38.143925 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.143930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.143947 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:53:38.144080 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.144083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.144133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.144245 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:38.145077 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.145626 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.146516 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.146753 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.146838 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.149562 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:38.149565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:38.149567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:38.149593 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:38.150869 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.151370 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.151374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.151376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.151467 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.151746 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.151750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.151751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.151940 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:38.152087 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.152406 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.152534 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:38.182905 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:38.173919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.16.40.193:36949)
0812 18:53:38.174086 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:38.174091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:38.174256 (+ 165us) server_negotiation.cc:407] Connection header received
0812 18:53:38.174445 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:38.174447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:38.174497 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:38.174591 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:38.175338 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.175857 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.176822 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:38.177054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:38.177169 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:38.179843 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:38.179847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:38.179849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:38.179877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:38.181308 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.181769 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.181772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.181773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.181851 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:38.182154 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:38.182157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:38.182158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:38.182338 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:38.182440 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:38.182672 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 18:53:38.182778 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 18:53:38.184839 17356 tablet_service.cc:2908] Rejecting scan request for tablet b1636998a6624a2dbc01e6127d1877f3: Uninitialized: clean time has not yet been initialized
I20260812 18:53:38.187774 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17303
I20260812 18:53:38.197845 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17467
I20260812 18:53:38.209407 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17625
I20260812 18:53:38.216949 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17190
2026-08-12T18:53:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4035 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17829](info): set up 2 sockets
Aug 12 18:53:38 dist-test-slave-rnrw krb5kdc[17829](info): commencing operation
krb5kdc: starting...
W20260812 18:53:39.361064 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:39 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:39.504526 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37587
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:37587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:39.626519 17842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:39.626832 17842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:39.626875 17842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:39.631060 17842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:39.631132 17842 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 18:53:39.631151 17842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:39.631170 17842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:39.631187 17842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:39.631203 17842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:39.636812 17842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:37587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:39.637943 17842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:39.639477 17842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:39.649132 17848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:39.649150 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:39.649166 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:39.649480 17842 server_base.cc:1034] running on GCE node
I20260812 18:53:39.650290 17842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:39.651571 17842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:39.652778 17842 hybrid_clock.cc:648] HybridClock initialized: now 1786560819652737 us; error 55 us; skew 500 ppm
Aug 12 18:53:39 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:39.656073 17842 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:39.657244 17842 webserver.cc:460] Webserver started at http://127.16.40.254:37373/ using document root <none> and password file <none>
I20260812 18:53:39.657879 17842 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:39.657989 17842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:39.658249 17842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:39.660491 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "307657da93d348b8b6e04c1db040b515"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "b56a24bf08941aa1beb555e3310308ad"
server_key_iv: "d1a73263e9c21f22eaed7c19e6cf762b"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.661180 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "307657da93d348b8b6e04c1db040b515"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "b56a24bf08941aa1beb555e3310308ad"
server_key_iv: "d1a73263e9c21f22eaed7c19e6cf762b"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.665489 17842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:39.668525 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:39.670329 17842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:39.670514 17842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "307657da93d348b8b6e04c1db040b515"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "b56a24bf08941aa1beb555e3310308ad"
server_key_iv: "d1a73263e9c21f22eaed7c19e6cf762b"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.670655 17842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:39.682636 17842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:39.686182 17842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:39.686429 17842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:39.695006 17842 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:37587
I20260812 18:53:39.695036 17926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:37587 every 8 connection(s)
I20260812 18:53:39.696038 17842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:39.699424 17927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:39.703112 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17842
I20260812 18:53:39.703269 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:39.703519 16547 external_mini_cluster.cc:1428] Setting key 9f400e9522be308b949f7fc91b292287
I20260812 18:53:39.706679 17927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Bootstrap starting.
I20260812 18:53:39.709396 17927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:39.710299 17927 log.cc:824] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:39.713188 17927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: No bootstrap required, opened a new log
Aug 12 18:53:39 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:39.718304 17927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } }
I20260812 18:53:39.718680 17927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:39.718747 17927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307657da93d348b8b6e04c1db040b515, State: Initialized, Role: FOLLOWER
I20260812 18:53:39.719328 17927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } }
I20260812 18:53:39.719501 17927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:39.719583 17927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:39.719686 17927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:39.720237 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:39.705241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.16.40.254:37587)
0812 18:53:39.705789 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:39.705814 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:39.705855 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:53:39.706515 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:39.706533 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:39.706943 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:39.707341 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:39.708230 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:39.709401 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:39.710296 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:39.710973 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:39.711184 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:39.714246 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:39.714254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:39.714275 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:39.714315 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:39.716414 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:39.717018 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:39.717026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:39.717033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:39.717205 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:39.717583 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:39.717588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:39.717590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:39.718210 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:39.718437 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:39.718782 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 18:53:39.719138 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":160,"threads_started":1}
I20260812 18:53:39.721060 17927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } }
I20260812 18:53:39.721455 17927 leader_election.cc:302] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307657da93d348b8b6e04c1db040b515; no voters:
I20260812 18:53:39.721869 17927 leader_election.cc:288] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:39.721982 17932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:39.722206 17932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [term 1 LEADER]: Becoming Leader. State: Replica: 307657da93d348b8b6e04c1db040b515, State: Running, Role: LEADER
I20260812 18:53:39.722662 17932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } }
I20260812 18:53:39.723276 17927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:39.724059 17932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307657da93d348b8b6e04c1db040b515. Latest consensus state: current_term: 1 leader_uuid: "307657da93d348b8b6e04c1db040b515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } } }
I20260812 18:53:39.724192 17932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:39.724498 17933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307657da93d348b8b6e04c1db040b515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307657da93d348b8b6e04c1db040b515" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37587 } } }
I20260812 18:53:39.724716 17933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:39.724836 17943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:39.728145 17943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:39.734153 17943 catalog_manager.cc:1347] Generated new cluster ID: 9303efab3d97420689d4fe42d6885c62
I20260812 18:53:39.734249 17943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:39.744913 17943 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:39.745970 17943 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:39.759115 17943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Generated new TSK 0
I20260812 18:53:39.759954 17943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:39.798139 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37587
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:39.915882 17957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:39.916206 17957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:39.916287 17957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:39.920354 17957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:39.920460 17957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:39.920578 17957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:39.920674 17957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:39.926513 17957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:39.927685 17957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:39.928785 17957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:39.939189 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:39.939216 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:39.939637 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:39.939718 17957 server_base.cc:1034] running on GCE node
I20260812 18:53:39.940230 17957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:39.941042 17957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:39.942281 17957 hybrid_clock.cc:648] HybridClock initialized: now 1786560819942270 us; error 35 us; skew 500 ppm
Aug 12 18:53:39 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:39.950351 17957 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:39.951412 17957 webserver.cc:460] Webserver started at http://127.16.40.193:45467/ using document root <none> and password file <none>
I20260812 18:53:39.952099 17957 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:39.952201 17957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:39.952450 17957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:39.954685 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "01e78f21b22448058e71c375777316ae"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "a6d46f3ac00bf50e75234a98e3115963"
server_key_iv: "dd9b851356b0ffc32450cd60dc07f41a"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.955317 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "01e78f21b22448058e71c375777316ae"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "a6d46f3ac00bf50e75234a98e3115963"
server_key_iv: "dd9b851356b0ffc32450cd60dc07f41a"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.959649 17957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 18:53:39.962527 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:39.964059 17957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:39.964236 17957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "01e78f21b22448058e71c375777316ae"
format_stamp: "Formatted at 2026-08-12 18:53:39 on dist-test-slave-rnrw"
server_key: "a6d46f3ac00bf50e75234a98e3115963"
server_key_iv: "dd9b851356b0ffc32450cd60dc07f41a"
server_key_version: "encryptionkey@0"
I20260812 18:53:39.964372 17957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:40.019894 17957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:40.024515 17957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:40.024801 17957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:40.026588 17957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:40.026660 17957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:40.026747 17957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:40.026785 17957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:40.039104 17957 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43237
I20260812 18:53:40.039191 18106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43237 every 8 connection(s)
I20260812 18:53:40.040155 17957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:40.042575 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.027693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:43379 (local address 127.16.40.254:37587)
0812 18:53:40.027873 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.027878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.028883 (+ 1005us) server_negotiation.cc:407] Connection header received
0812 18:53:40.030341 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.030347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.030419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.030501 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.032693 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.033343 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.034896 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.035153 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.035278 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.038151 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.038155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.038157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.038185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.040135 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.040841 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.040846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.040848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.040963 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.041381 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.041384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.041385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.041687 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.041780 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.042305 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.042444 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:53:40.043440 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.028089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.193:43379)
0812 18:53:40.028704 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.028767 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.030035 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.031003 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.031015 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.031570 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.032380 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.032405 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.033498 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.033502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.034753 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.034764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.035341 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.035345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.035522 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.036177 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.036195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.037932 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.040282 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.040289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.040306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.040701 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.041085 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.041089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.041092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.041269 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.041797 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.041802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.042059 (+ 257us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.042224 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.042502 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":112,"threads_started":1}
I20260812 18:53:40.044914 18107 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37587
I20260812 18:53:40.045089 18107 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:40.045722 18107 heartbeater.cc:511] Master 127.16.40.254:37587 requested a full tablet report, sending...
I20260812 18:53:40.047549 17877 ts_manager.cc:194] Registered new tserver with Master: 01e78f21b22448058e71c375777316ae (127.16.40.193:43237)
I20260812 18:53:40.048755 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 17957
I20260812 18:53:40.048884 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:40.049008 17877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:43379
I20260812 18:53:40.049220 16547 external_mini_cluster.cc:1428] Setting key 8cfe4510ea21df245f0960b2c93b7349
I20260812 18:53:40.057147 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.047440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:60175 (local address 127.16.40.254:37587)
0812 18:53:40.047567 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.047571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.048850 (+ 1279us) server_negotiation.cc:407] Connection header received
0812 18:53:40.049058 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.049062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.049124 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.049214 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.049974 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.050749 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.051917 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.052138 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.052224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.053594 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.053601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.053607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.053647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.055256 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.055804 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.055806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.055807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.055887 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.056242 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.056244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.056245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.056427 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.056506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.056901 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.057013 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:53:40.057343 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.047340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.193:60175)
0812 18:53:40.048763 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.048779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.048936 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.049350 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.049354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.049595 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.049751 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.049759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.050879 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.050883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.051776 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.051792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.052263 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.052269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.052404 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.053065 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.053077 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.053436 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.055382 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.055389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.055394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.055695 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.055988 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.055994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.055998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.056141 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.056531 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.056537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.056695 (+ 158us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.056829 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.057026 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:40.084597 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37587
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:40.206034 18116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:40.206362 18116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:40.206413 18116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:40.210681 18116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:40.210775 18116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:40.210816 18116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:40.210851 18116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:40.216571 18116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:40.217890 18116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:40.218950 18116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:40.229509 18125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:40.229518 18127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:40.229648 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:40.229859 18116 server_base.cc:1034] running on GCE node
I20260812 18:53:40.230392 18116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:40.231130 18116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:40.232327 18116 hybrid_clock.cc:648] HybridClock initialized: now 1786560820232310 us; error 37 us; skew 500 ppm
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:40.235467 18116 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:40.236483 18116 webserver.cc:460] Webserver started at http://127.16.40.194:42621/ using document root <none> and password file <none>
I20260812 18:53:40.237142 18116 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:40.237211 18116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:40.237396 18116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:40.239571 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "94e51d5ff9df422fae0c5b0068b2ad39"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "8f19524ee07aa770ac0ec4163ccf6e9e"
server_key_iv: "c732ca3c29e6713c48bf58f34f799125"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.240118 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "94e51d5ff9df422fae0c5b0068b2ad39"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "8f19524ee07aa770ac0ec4163ccf6e9e"
server_key_iv: "c732ca3c29e6713c48bf58f34f799125"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.244617 18116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:53:40.247725 18137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:40.249300 18116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:40.249459 18116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "94e51d5ff9df422fae0c5b0068b2ad39"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "8f19524ee07aa770ac0ec4163ccf6e9e"
server_key_iv: "c732ca3c29e6713c48bf58f34f799125"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.249671 18116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:40.262203 18116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:40.266126 18116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:40.266379 18116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:40.268188 18116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:40.268247 18116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:40.268325 18116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:40.268360 18116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:40.280593 18116 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:37689
I20260812 18:53:40.280663 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:37689 every 8 connection(s)
I20260812 18:53:40.281870 18116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:40.283150 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 18116
I20260812 18:53:40.283268 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:40.283545 16547 external_mini_cluster.cc:1428] Setting key a5337864ca508d5a8624ee3c16e544b4
I20260812 18:53:40.285308 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.269521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53075 (local address 127.16.40.254:37587)
0812 18:53:40.269697 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.269704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.270770 (+ 1066us) server_negotiation.cc:407] Connection header received
0812 18:53:40.272043 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.272048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.272122 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.272244 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.273996 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.274713 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.276355 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.276759 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.276909 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.280378 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.280383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.280388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.280429 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.282411 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.283171 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.283176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.283179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.283285 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.283792 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.283796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.283797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.284071 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.284200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.284966 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.285139 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:53:40.286765 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.269982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.194:53075)
0812 18:53:40.270568 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.270628 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.271769 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.272471 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.272479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.273154 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.273801 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.273819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.274873 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.274877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.276201 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.276214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.276966 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.276971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.277189 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.277962 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.277983 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.280152 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.282576 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.282587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.282608 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.283008 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.283471 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.283475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.283476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.283697 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.284195 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.284201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.284663 (+ 462us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.284905 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.285340 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":177,"threads_started":1}
I20260812 18:53:40.288177 18268 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37587
I20260812 18:53:40.288383 18268 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:40.289019 18268 heartbeater.cc:511] Master 127.16.40.254:37587 requested a full tablet report, sending...
I20260812 18:53:40.290241 17877 ts_manager.cc:194] Registered new tserver with Master: 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194:37689)
I20260812 18:53:40.291144 17877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:53075
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:53:40.300027 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.290874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.194:57015)
0812 18:53:40.291179 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.291192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.291315 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.291715 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.291718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.291940 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.292086 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.292095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.293356 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.293360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.294089 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.294101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.294742 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.294745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.294818 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.295424 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.295436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.295808 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.297859 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.297861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.297863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.298095 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.298347 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.298349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.298351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.298430 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.299331 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.299334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.299493 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.299606 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.299817 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
I20260812 18:53:40.300081 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.290770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:57015 (local address 127.16.40.254:37587)
0812 18:53:40.290907 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.290911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.291269 (+ 358us) server_negotiation.cc:407] Connection header received
0812 18:53:40.291421 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.291424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.291469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.291562 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.292230 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.293176 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.294225 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.294610 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.294742 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.295983 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.295990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.295994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.296023 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.297749 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.298185 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.298188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.298189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.298266 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.298729 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.298733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.298734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.299040 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.299153 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.299687 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.299847 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:40.320346 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37587
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:40.440936 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:40.441226 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:40.441272 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:40.445616 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:40.445708 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:40.445757 18277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:40.445792 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:40.451458 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:40.452612 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:40.453776 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:40.464215 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:40.464243 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:40.464416 18277 server_base.cc:1034] running on GCE node
W20260812 18:53:40.464232 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:40.465050 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:40.465792 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:40.467002 18277 hybrid_clock.cc:648] HybridClock initialized: now 1786560820466984 us; error 32 us; skew 500 ppm
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:40.470233 18277 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:40.471192 18277 webserver.cc:460] Webserver started at http://127.16.40.195:42327/ using document root <none> and password file <none>
I20260812 18:53:40.471802 18277 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:40.471897 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:40.472149 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:40.474427 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "7254da3822204c4290b6eb78f491ea38"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "b0319737d29d7a15f317943cf7f4e84e"
server_key_iv: "a913de9436f004d7c65b40ebda4fd592"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.475033 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "7254da3822204c4290b6eb78f491ea38"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "b0319737d29d7a15f317943cf7f4e84e"
server_key_iv: "a913de9436f004d7c65b40ebda4fd592"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.479565 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 18:53:40.482395 18295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:40.483742 18277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:40.484016 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "7254da3822204c4290b6eb78f491ea38"
format_stamp: "Formatted at 2026-08-12 18:53:40 on dist-test-slave-rnrw"
server_key: "b0319737d29d7a15f317943cf7f4e84e"
server_key_iv: "a913de9436f004d7c65b40ebda4fd592"
server_key_version: "encryptionkey@0"
I20260812 18:53:40.484148 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:40.499902 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:40.503693 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:40.503933 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:40.505759 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:40.505848 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:40.505926 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:40.505973 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:40.518329 18277 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:43997
I20260812 18:53:40.518424 18440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:43997 every 8 connection(s)
I20260812 18:53:40.519574 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:40.522270 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.506672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:54135 (local address 127.16.40.254:37587)
0812 18:53:40.506864 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.506869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.508267 (+ 1398us) server_negotiation.cc:407] Connection header received
0812 18:53:40.509462 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.509466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.509533 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.509630 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.511362 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.512252 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.513828 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.514080 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.514676 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.518022 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.518027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.518030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.518068 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.519870 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.520629 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.520633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.520635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.520750 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.521150 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.521152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.521154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.521358 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.521438 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.522013 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.522140 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 18:53:40.523288 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.507253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.195:54135)
0812 18:53:40.508080 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.508139 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.509197 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.509843 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.509851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.510420 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.511185 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.511211 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.512403 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.512407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.513683 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.513694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.514683 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.514688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.514998 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.515753 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.515782 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.517745 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.520011 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.520021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.520043 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.520447 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.520854 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.520858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.520862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.521045 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.521462 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.521468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.521758 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.521926 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.522234 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":165,"threads_started":1}
I20260812 18:53:40.524850 18441 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37587
I20260812 18:53:40.525022 18441 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:40.525740 18441 heartbeater.cc:511] Master 127.16.40.254:37587 requested a full tablet report, sending...
I20260812 18:53:40.527140 17878 ts_manager.cc:194] Registered new tserver with Master: 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997)
I20260812 18:53:40.528053 17878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:54135
I20260812 18:53:40.529139 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 18277
I20260812 18:53:40.529279 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:40.529608 16547 external_mini_cluster.cc:1428] Setting key 9a1bbd1df8b7503fd93dbe16dddec264
I20260812 18:53:40.532379 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:53:40.536362 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.527628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:33515 (local address 127.16.40.254:37587)
0812 18:53:40.527785 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.527790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.528027 (+ 237us) server_negotiation.cc:407] Connection header received
0812 18:53:40.528184 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.528189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.528242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.528313 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.528981 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.529626 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.531035 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.531379 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.531510 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.532924 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.532928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.532931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.532962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.534554 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.535050 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.535053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.535054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.535136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.535432 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.535437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.535439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.535646 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.535761 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.536050 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.536183 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:53:40.536376 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.527552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.16.40.195:33515)
0812 18:53:40.527931 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.527948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.528069 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.528373 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.528382 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.528696 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:40.528847 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.528855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.529803 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.529806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.530886 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.530900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.531506 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.531509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.531593 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.532189 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:40.532205 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:40.532729 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:40.534688 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.534693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.534695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.534919 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.535235 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:40.535238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:40.535240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:40.535336 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:40.535749 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:40.535753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:40.535865 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.535982 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.536174 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":163,"threads_started":1}
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 18:53:40 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:40.562700 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.553952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.16.40.254:37587)
0812 18:53:40.554118 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.554122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.554134 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:53:40.554177 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.554179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.554219 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.554322 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:40.555214 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.555744 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.556660 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.556875 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.556990 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.559484 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:40.559487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:40.559489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:40.559519 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:40.561084 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.561514 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.561518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.561519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.561600 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:40.561845 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:40.561847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:40.561848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:40.562048 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:40.562181 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.562461 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.562566 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 18:53:40.566073 17878 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:40.569460 17878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:40.585376 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.579777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37689 (local address 127.0.0.1:46136)
0812 18:53:40.580287 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.580299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.580375 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.581111 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.581114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.581132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.581230 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.581239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582474 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.582479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.583665 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.583677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.584761 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.584765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.584882 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.584945 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.585040 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.585179 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":152,"thread_start_us":101,"threads_started":1}
I20260812 18:53:40.586095 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.580238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.16.40.194:37689)
0812 18:53:40.580664 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.580676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.580698 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:53:40.580770 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.580778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.580947 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.581114 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.581362 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582333 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.583834 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.584557 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.585606 (+ 1049us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.585769 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.585824 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.585948 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":109,"threads_started":1}
I20260812 18:53:40.586094 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.578971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:43997 (local address 127.0.0.1:59984)
0812 18:53:40.579751 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.579811 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.579951 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.580616 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.580622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.580648 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.580854 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.580861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582430 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.582434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.584205 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.584217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.585671 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.585675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.585780 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.585810 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.585892 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.585969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":218,"thread_start_us":125,"threads_started":1}
I20260812 18:53:40.586383 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.579200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43237 (local address 127.0.0.1:42568)
0812 18:53:40.580179 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.580196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.580297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.580862 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.580865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.580884 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.580995 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.581002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582430 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.582434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.584205 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.584217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.585783 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.585787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.585897 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.585923 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.585988 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.586059 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":204,"thread_start_us":103,"threads_started":1}
I20260812 18:53:40.586758 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.579721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.16.40.193:43237)
0812 18:53:40.580156 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.580163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.580407 (+ 244us) server_negotiation.cc:407] Connection header received
0812 18:53:40.580499 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.580504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.580722 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.580870 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.581136 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582279 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.584361 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.585187 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.585292 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.586146 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.586532 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.586636 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":84,"threads_started":1}
I20260812 18:53:40.587092 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.579154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.16.40.195:43997)
0812 18:53:40.579908 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.579921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.579949 (+ 28us) server_negotiation.cc:407] Connection header received
0812 18:53:40.580183 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.580190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.580420 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.580632 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.580998 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.582279 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.584355 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.585284 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.585414 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.585604 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.585938 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.586055 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":134,"threads_started":1}
I20260812 18:53:40.589524 18193 tablet_service.cc:1467] Processing CreateTablet for tablet fa36bbd0d16742938e6598417e0b25a5 (DEFAULT_TABLE table=test-table [id=784e87804077481786df36678234e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.589514 18032 tablet_service.cc:1467] Processing CreateTablet for tablet fa36bbd0d16742938e6598417e0b25a5 (DEFAULT_TABLE table=test-table [id=784e87804077481786df36678234e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.590543 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa36bbd0d16742938e6598417e0b25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.590581 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa36bbd0d16742938e6598417e0b25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.592015 18355 tablet_service.cc:1467] Processing CreateTablet for tablet fa36bbd0d16742938e6598417e0b25a5 (DEFAULT_TABLE table=test-table [id=784e87804077481786df36678234e8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.593171 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa36bbd0d16742938e6598417e0b25a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.598613 18470 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Bootstrap starting.
I20260812 18:53:40.598632 18469 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Bootstrap starting.
I20260812 18:53:40.600808 18471 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Bootstrap starting.
I20260812 18:53:40.601061 18470 tablet_bootstrap.cc:654] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.601222 18469 tablet_bootstrap.cc:654] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.602048 18470 log.cc:824] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:40.602048 18469 log.cc:824] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:40.603052 18471 tablet_bootstrap.cc:654] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.603955 18471 log.cc:824] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:40.604306 18469 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: No bootstrap required, opened a new log
I20260812 18:53:40.604530 18469 ts_tablet_manager.cc:1397] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 18:53:40.604996 18470 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: No bootstrap required, opened a new log
I20260812 18:53:40.605209 18470 ts_tablet_manager.cc:1397] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:53:40.605870 18471 tablet_bootstrap.cc:492] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: No bootstrap required, opened a new log
I20260812 18:53:40.606084 18471 ts_tablet_manager.cc:1397] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 18:53:40.608935 18471 raft_consensus.cc:348] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.608942 18469 raft_consensus.cc:348] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.609198 18471 raft_consensus.cc:374] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.609206 18469 raft_consensus.cc:374] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.609258 18471 raft_consensus.cc:729] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7254da3822204c4290b6eb78f491ea38, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.609261 18469 raft_consensus.cc:729] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e51d5ff9df422fae0c5b0068b2ad39, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.609716 18471 consensus_queue.cc:260] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.609746 18469 consensus_queue.cc:260] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.609680 18470 raft_consensus.cc:348] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.609920 18470 raft_consensus.cc:374] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.609968 18470 raft_consensus.cc:729] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e78f21b22448058e71c375777316ae, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.610396 18470 consensus_queue.cc:260] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.610553 18441 heartbeater.cc:503] Master 127.16.40.254:37587 was elected leader, sending a full tablet report...
I20260812 18:53:40.610891 18469 ts_tablet_manager.cc:1428] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260812 18:53:40.611234 18268 heartbeater.cc:503] Master 127.16.40.254:37587 was elected leader, sending a full tablet report...
I20260812 18:53:40.611455 18471 ts_tablet_manager.cc:1428] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 18:53:40.612325 18107 heartbeater.cc:503] Master 127.16.40.254:37587 was elected leader, sending a full tablet report...
I20260812 18:53:40.612594 18470 ts_tablet_manager.cc:1428] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 18:53:40.639616 18477 raft_consensus.cc:482] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:40.639828 18477 raft_consensus.cc:504] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.641103 18477 leader_election.cc:288] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997), 01e78f21b22448058e71c375777316ae (127.16.40.193:43237)
I20260812 18:53:40.645561 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.641570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53831 (local address 127.16.40.195:43997)
0812 18:53:40.641721 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.641727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.641744 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:53:40.641803 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.641806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.641862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.641964 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.642268 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.642938 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.644298 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.644942 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.645076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.645132 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.645295 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.645393 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 18:53:40.645615 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.641364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:43997 (local address 127.16.40.194:53831)
0812 18:53:40.641480 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.641500 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.641615 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.641977 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.641980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.642032 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.642151 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.642157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.643064 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.643068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.644157 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.644168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.645064 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.645067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.645169 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.645192 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.645259 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.645398 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"spinlock_wait_cycles":43008}
I20260812 18:53:40.646404 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa36bbd0d16742938e6598417e0b25a5" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7254da3822204c4290b6eb78f491ea38" is_pre_election: true
I20260812 18:53:40.646782 18380 raft_consensus.cc:2454] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 0.
I20260812 18:53:40.646844 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.641544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:55489 (local address 127.16.40.193:43237)
0812 18:53:40.641780 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.641785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.641979 (+ 194us) server_negotiation.cc:407] Connection header received
0812 18:53:40.642160 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.642162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.642204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.642293 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.642566 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.643319 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.645516 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.646168 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.646292 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.646339 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.646655 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.646735 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 18:53:40.646867 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.641466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43237 (local address 127.16.40.194:55489)
0812 18:53:40.641888 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.641898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.642045 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.642304 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.642307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.642323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.642455 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.642460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.643446 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.643452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.645386 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.645397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.646331 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.646338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.646445 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.646470 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.646638 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.646715 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"spinlock_wait_cycles":6528,"thread_start_us":171,"threads_started":1}
I20260812 18:53:40.647329 18138 leader_election.cc:302] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7254da3822204c4290b6eb78f491ea38, 94e51d5ff9df422fae0c5b0068b2ad39; no voters:
I20260812 18:53:40.647423 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa36bbd0d16742938e6598417e0b25a5" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae" is_pre_election: true
I20260812 18:53:40.647676 18477 raft_consensus.cc:2781] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:40.647775 18477 raft_consensus.cc:482] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:40.647746 18054 raft_consensus.cc:2454] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 0.
I20260812 18:53:40.647811 18477 raft_consensus.cc:3037] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.649163 18477 raft_consensus.cc:504] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.649592 18477 leader_election.cc:288] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 election: Requested vote from peers 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997), 01e78f21b22448058e71c375777316ae (127.16.40.193:43237)
I20260812 18:53:40.649960 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa36bbd0d16742938e6598417e0b25a5" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7254da3822204c4290b6eb78f491ea38"
I20260812 18:53:40.650034 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa36bbd0d16742938e6598417e0b25a5" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae"
I20260812 18:53:40.650136 18380 raft_consensus.cc:3037] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.650163 18054 raft_consensus.cc:3037] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.651265 18054 raft_consensus.cc:2454] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 1.
I20260812 18:53:40.651419 18380 raft_consensus.cc:2454] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 1.
I20260812 18:53:40.651607 18139 leader_election.cc:302] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e78f21b22448058e71c375777316ae, 94e51d5ff9df422fae0c5b0068b2ad39; no voters:
I20260812 18:53:40.651808 18477 raft_consensus.cc:2781] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:40.652045 18477 raft_consensus.cc:686] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 LEADER]: Becoming Leader. State: Replica: 94e51d5ff9df422fae0c5b0068b2ad39, State: Running, Role: LEADER
I20260812 18:53:40.652420 18477 consensus_queue.cc:237] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } }
I20260812 18:53:40.656167 17878 catalog_manager.cc:5495] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } health_report { overall_health: UNKNOWN } } }
I20260812 18:53:40.676947 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.672653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.16.40.194:37689)
0812 18:53:40.672838 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.672843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.672860 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:53:40.673128 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.673131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.673195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.673308 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:40.673644 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.674175 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.675117 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.675402 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.675515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.675901 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:40.676444 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:40.676619 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.676701 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.676768 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 18:53:40.684123 18380 raft_consensus.cc:1261] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Refusing update from remote peer 94e51d5ff9df422fae0c5b0068b2ad39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:40.684942 18477 consensus_queue.cc:1035] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:40.686028 18054 raft_consensus.cc:1261] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Refusing update from remote peer 94e51d5ff9df422fae0c5b0068b2ad39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:40.686802 18477 consensus_queue.cc:1035] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:40.697512 18484 mvcc.cc:205] Tried to move back new op lower bound from 7317753121512423424 to 7317753121395712000. Current Snapshot: MvccSnapshot[applied={T|T < 7317753121512423424 or (T in {7317753121512423424})}]
I20260812 18:53:40.708603 17877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:40.708846 17877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:40.711598 17877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Sending DeleteTablet for 3 replicas of tablet fa36bbd0d16742938e6598417e0b25a5
I20260812 18:53:40.712749 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet fa36bbd0d16742938e6598417e0b25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:59984
I20260812 18:53:40.713379 18498 tablet_replica.cc:330] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: stopping tablet replica
I20260812 18:53:40.713405 18032 tablet_service.cc:1514] Processing DeleteTablet for tablet fa36bbd0d16742938e6598417e0b25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:42568
I20260812 18:53:40.713912 18499 tablet_replica.cc:330] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: stopping tablet replica
I20260812 18:53:40.713912 18498 raft_consensus.cc:2229] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:40.714449 18498 raft_consensus.cc:2258] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:40.714403 18193 tablet_service.cc:1514] Processing DeleteTablet for tablet fa36bbd0d16742938e6598417e0b25a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46136
I20260812 18:53:40.714913 18500 tablet_replica.cc:330] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: stopping tablet replica
I20260812 18:53:40.715175 18499 raft_consensus.cc:2229] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:40.715221 18500 raft_consensus.cc:2229] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:40.715194 17877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 18:53:40.715482 18499 raft_consensus.cc:2258] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:40.715652 18500 raft_consensus.cc:2258] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 18:53:40.715685 17877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:40.717585 18500 ts_tablet_manager.cc:1905] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:40.718612 18498 ts_tablet_manager.cc:1905] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:40.720803 18500 ts_tablet_manager.cc:1918] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:53:40.720995 18500 log.cc:1196] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/fa36bbd0d16742938e6598417e0b25a5
I20260812 18:53:40.721405 18500 ts_tablet_manager.cc:1939] T fa36bbd0d16742938e6598417e0b25a5 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting consensus metadata
I20260812 18:53:40.721530 18499 ts_tablet_manager.cc:1905] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:40.722324 18032 tablet_service.cc:1467] Processing CreateTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (DEFAULT_TABLE table=test-table [id=5d6dbb69db444604b48910dd61474427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.722644 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.722921 17861 catalog_manager.cc:4841] TS 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194:37689): tablet fa36bbd0d16742938e6598417e0b25a5 (table test-table [id=784e87804077481786df36678234e8f5]) successfully deleted
I20260812 18:53:40.723533 18355 tablet_service.cc:1467] Processing CreateTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (DEFAULT_TABLE table=test-table [id=5d6dbb69db444604b48910dd61474427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.723817 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.725358 18470 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Bootstrap starting.
I20260812 18:53:40.726357 18471 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Bootstrap starting.
I20260812 18:53:40.726436 18470 tablet_bootstrap.cc:654] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.726574 18193 tablet_service.cc:1467] Processing CreateTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (DEFAULT_TABLE table=test-table [id=5d6dbb69db444604b48910dd61474427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:40.726846 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:40.727466 18471 tablet_bootstrap.cc:654] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.729002 18471 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: No bootstrap required, opened a new log
I20260812 18:53:40.729100 18471 ts_tablet_manager.cc:1397] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:40.729261 18469 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Bootstrap starting.
I20260812 18:53:40.729576 18471 raft_consensus.cc:348] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.729717 18471 raft_consensus.cc:374] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.729756 18471 raft_consensus.cc:729] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7254da3822204c4290b6eb78f491ea38, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.729903 18471 consensus_queue.cc:260] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.730197 18471 ts_tablet_manager.cc:1428] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:40.730406 18469 tablet_bootstrap.cc:654] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:40.732290 18499 ts_tablet_manager.cc:1918] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:53:40.733204 18469 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: No bootstrap required, opened a new log
I20260812 18:53:40.733320 18469 ts_tablet_manager.cc:1397] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260812 18:53:40.733349 18498 ts_tablet_manager.cc:1918] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:53:40.733568 18498 log.cc:1196] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/fa36bbd0d16742938e6598417e0b25a5
I20260812 18:53:40.733889 18469 raft_consensus.cc:348] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.733958 18498 ts_tablet_manager.cc:1939] T fa36bbd0d16742938e6598417e0b25a5 P 7254da3822204c4290b6eb78f491ea38: Deleting consensus metadata
I20260812 18:53:40.734028 18469 raft_consensus.cc:374] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.734071 18469 raft_consensus.cc:729] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e51d5ff9df422fae0c5b0068b2ad39, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.734239 18469 consensus_queue.cc:260] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.735369 18499 log.cc:1196] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/fa36bbd0d16742938e6598417e0b25a5
I20260812 18:53:40.735381 17861 catalog_manager.cc:4841] TS 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997): tablet fa36bbd0d16742938e6598417e0b25a5 (table test-table [id=784e87804077481786df36678234e8f5]) successfully deleted
I20260812 18:53:40.735741 18499 ts_tablet_manager.cc:1939] T fa36bbd0d16742938e6598417e0b25a5 P 01e78f21b22448058e71c375777316ae: Deleting consensus metadata
I20260812 18:53:40.734820 18470 tablet_bootstrap.cc:492] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: No bootstrap required, opened a new log
I20260812 18:53:40.736922 18470 ts_tablet_manager.cc:1397] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260812 18:53:40.737214 17862 catalog_manager.cc:4841] TS 01e78f21b22448058e71c375777316ae (127.16.40.193:43237): tablet fa36bbd0d16742938e6598417e0b25a5 (table test-table [id=784e87804077481786df36678234e8f5]) successfully deleted
I20260812 18:53:40.737430 18469 ts_tablet_manager.cc:1428] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260812 18:53:40.737844 18470 raft_consensus.cc:348] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.737986 18470 raft_consensus.cc:374] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:40.738024 18470 raft_consensus.cc:729] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e78f21b22448058e71c375777316ae, State: Initialized, Role: FOLLOWER
I20260812 18:53:40.738199 18470 consensus_queue.cc:260] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.738585 18470 ts_tablet_manager.cc:1428] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260812 18:53:40.770713 18442 tablet.cc:2369] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:53:40.783296 18269 tablet.cc:2369] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:53:40.792133 18108 tablet.cc:2369] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:53:40.976428 18475 raft_consensus.cc:482] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:40.976684 18475 raft_consensus.cc:504] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.977890 18475 leader_election.cc:288] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e78f21b22448058e71c375777316ae (127.16.40.193:43237), 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194:37689)
I20260812 18:53:40.981736 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.978270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:39885 (local address 127.16.40.193:43237)
0812 18:53:40.978412 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.978418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.978438 (+ 20us) server_negotiation.cc:407] Connection header received
0812 18:53:40.978645 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.978653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.978700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.978790 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.979080 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.979581 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.980707 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.981151 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.981286 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.981343 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.981532 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.981609 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:53:40.981775 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.978136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43237 (local address 127.16.40.195:39885)
0812 18:53:40.978366 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.978384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.978489 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.978804 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.978807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.978826 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.978953 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.978958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.979697 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.979702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.980572 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.980582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.981286 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.981290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.981405 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.981423 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.981512 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.981598 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":27}
I20260812 18:53:40.982312 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.978365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:45939 (local address 127.16.40.194:37689)
0812 18:53:40.978485 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:40.978490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:40.978704 (+ 214us) server_negotiation.cc:407] Connection header received
0812 18:53:40.978889 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:40.978900 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:40.978951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:40.979048 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:40.979360 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.979896 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.981188 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.981649 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.981795 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.981849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:40.982112 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 18:53:40.982197 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:53:40.982311 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:40.978239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37689 (local address 127.16.40.195:45939)
0812 18:53:40.978581 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:40.978594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:40.978753 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:40.979079 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:40.979082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:40.979100 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:40.979233 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.979239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.980023 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.980030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.981048 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:40.981059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:40.981785 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:40.981789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:40.981904 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:40.981938 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 18:53:40.982066 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 18:53:40.982146 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260812 18:53:40.982300 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bf73e5fd24bd5b2c6b8973a8d1bb4" candidate_uuid: "7254da3822204c4290b6eb78f491ea38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae" is_pre_election: true
I20260812 18:53:40.982493 18054 raft_consensus.cc:2454] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7254da3822204c4290b6eb78f491ea38 in term 0.
I20260812 18:53:40.982913 18298 leader_election.cc:302] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e78f21b22448058e71c375777316ae, 7254da3822204c4290b6eb78f491ea38; no voters:
I20260812 18:53:40.982936 18214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bf73e5fd24bd5b2c6b8973a8d1bb4" candidate_uuid: "7254da3822204c4290b6eb78f491ea38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" is_pre_election: true
I20260812 18:53:40.983172 18214 raft_consensus.cc:2454] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7254da3822204c4290b6eb78f491ea38 in term 0.
I20260812 18:53:40.983210 18475 raft_consensus.cc:2781] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:40.983263 18475 raft_consensus.cc:482] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:40.983299 18475 raft_consensus.cc:3037] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.984414 18475 raft_consensus.cc:504] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.984817 18475 leader_election.cc:288] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [CANDIDATE]: Term 1 election: Requested vote from peers 01e78f21b22448058e71c375777316ae (127.16.40.193:43237), 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194:37689)
I20260812 18:53:40.985212 18214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bf73e5fd24bd5b2c6b8973a8d1bb4" candidate_uuid: "7254da3822204c4290b6eb78f491ea38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e51d5ff9df422fae0c5b0068b2ad39"
I20260812 18:53:40.985206 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94bf73e5fd24bd5b2c6b8973a8d1bb4" candidate_uuid: "7254da3822204c4290b6eb78f491ea38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae"
I20260812 18:53:40.985343 18214 raft_consensus.cc:3037] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.985342 18054 raft_consensus.cc:3037] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:40.986532 18214 raft_consensus.cc:2454] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7254da3822204c4290b6eb78f491ea38 in term 1.
I20260812 18:53:40.986533 18054 raft_consensus.cc:2454] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7254da3822204c4290b6eb78f491ea38 in term 1.
I20260812 18:53:40.986855 18298 leader_election.cc:302] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e78f21b22448058e71c375777316ae, 7254da3822204c4290b6eb78f491ea38; no voters:
I20260812 18:53:40.987069 18475 raft_consensus.cc:2781] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:40.987241 18475 raft_consensus.cc:686] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 1 LEADER]: Becoming Leader. State: Replica: 7254da3822204c4290b6eb78f491ea38, State: Running, Role: LEADER
I20260812 18:53:40.987535 18475 consensus_queue.cc:237] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:40.990082 17877 catalog_manager.cc:5495] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7254da3822204c4290b6eb78f491ea38 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "7254da3822204c4290b6eb78f491ea38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } health_report { overall_health: HEALTHY } } }
Aug 12 18:53:41 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560819, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:41.078771 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.066416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37587 (local address 127.0.0.1:33850)
0812 18:53:41.066786 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:41.066805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:41.066930 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:41.067267 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:41.067269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:41.067683 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:41.067782 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.067788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.068641 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.068644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.069287 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.069303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.069722 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.069725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.069787 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.073622 (+ 3835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:41.073635 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:41.075299 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:41.077213 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:41.077223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:41.077229 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:41.077471 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:41.077750 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:41.077752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:41.077753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:41.077837 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:41.078215 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:41.078221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:41.078344 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 18:53:41.078440 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 18:53:41.078569 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260812 18:53:41.078789 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.066644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.16.40.254:37587)
0812 18:53:41.066957 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.066965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.067001 (+ 36us) server_negotiation.cc:407] Connection header received
0812 18:53:41.067041 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.067044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.067082 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.067214 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:41.067898 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.068484 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.069394 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.069622 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.069702 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.075453 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:41.075457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:41.075459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:41.075482 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:41.077103 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.077580 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.077583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.077584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.077666 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.077919 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.077922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.077923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.078119 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:41.078207 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.078502 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.078632 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":132,"threads_started":1}
I20260812 18:53:41.083457 17877 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:33850:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 18:53:41.084326 17877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:41.090123 18032 tablet_service.cc:1467] Processing CreateTablet for tablet c6be00391fd343c5bcd343cef37a9f99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec2c15c50d574bed80f417ddf3f4aa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:41.090271 18193 tablet_service.cc:1467] Processing CreateTablet for tablet c6be00391fd343c5bcd343cef37a9f99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec2c15c50d574bed80f417ddf3f4aa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:41.090685 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6be00391fd343c5bcd343cef37a9f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:41.090698 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6be00391fd343c5bcd343cef37a9f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:41.093082 18355 tablet_service.cc:1467] Processing CreateTablet for tablet c6be00391fd343c5bcd343cef37a9f99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec2c15c50d574bed80f417ddf3f4aa46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:41.093539 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6be00391fd343c5bcd343cef37a9f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:41.093766 18470 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: Bootstrap starting.
I20260812 18:53:41.094583 18470 tablet_bootstrap.cc:654] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:41.095933 18469 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: Bootstrap starting.
I20260812 18:53:41.096916 18469 tablet_bootstrap.cc:654] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:41.096925 18470 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: No bootstrap required, opened a new log
I20260812 18:53:41.097049 18470 ts_tablet_manager.cc:1397] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 18:53:41.097471 18470 raft_consensus.cc:348] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.097556 18471 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: Bootstrap starting.
I20260812 18:53:41.097613 18470 raft_consensus.cc:374] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:41.097661 18470 raft_consensus.cc:729] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e78f21b22448058e71c375777316ae, State: Initialized, Role: FOLLOWER
I20260812 18:53:41.097798 18470 consensus_queue.cc:260] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.098129 18470 ts_tablet_manager.cc:1428] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 18:53:41.098446 18478 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.098541 18471 tablet_bootstrap.cc:654] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:41.098611 18478 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.099658 18469 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: No bootstrap required, opened a new log
I20260812 18:53:41.099774 18469 ts_tablet_manager.cc:1397] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 18:53:41.099943 18471 tablet_bootstrap.cc:492] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: No bootstrap required, opened a new log
I20260812 18:53:41.100046 18471 ts_tablet_manager.cc:1397] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:41.100291 18469 raft_consensus.cc:348] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.100426 18469 raft_consensus.cc:374] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:41.100467 18469 raft_consensus.cc:729] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e51d5ff9df422fae0c5b0068b2ad39, State: Initialized, Role: FOLLOWER
I20260812 18:53:41.100581 18471 raft_consensus.cc:348] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.100643 18469 consensus_queue.cc:260] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.100732 18471 raft_consensus.cc:374] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:41.100776 18471 raft_consensus.cc:729] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7254da3822204c4290b6eb78f491ea38, State: Initialized, Role: FOLLOWER
I20260812 18:53:41.100912 18471 consensus_queue.cc:260] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.100919 18495 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.101079 18495 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.101182 18471 ts_tablet_manager.cc:1428] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:41.101387 18475 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.101506 18475 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.101748 18469 ts_tablet_manager.cc:1428] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 18:53:41.168200 18495 raft_consensus.cc:482] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:41.168357 18495 raft_consensus.cc:504] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.168888 18495 leader_election.cc:288] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01e78f21b22448058e71c375777316ae (127.16.40.193:43237), 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997)
I20260812 18:53:41.169399 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6be00391fd343c5bcd343cef37a9f99" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7254da3822204c4290b6eb78f491ea38" is_pre_election: true
I20260812 18:53:41.169422 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6be00391fd343c5bcd343cef37a9f99" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae" is_pre_election: true
I20260812 18:53:41.169576 18380 raft_consensus.cc:2454] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 0.
I20260812 18:53:41.169601 18054 raft_consensus.cc:2454] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 0.
I20260812 18:53:41.169925 18138 leader_election.cc:302] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7254da3822204c4290b6eb78f491ea38, 94e51d5ff9df422fae0c5b0068b2ad39; no voters:
I20260812 18:53:41.170135 18495 raft_consensus.cc:2781] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:41.170203 18495 raft_consensus.cc:482] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:41.170243 18495 raft_consensus.cc:3037] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:41.171360 18495 raft_consensus.cc:504] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.171716 18495 leader_election.cc:288] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 election: Requested vote from peers 01e78f21b22448058e71c375777316ae (127.16.40.193:43237), 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997)
I20260812 18:53:41.172098 18054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6be00391fd343c5bcd343cef37a9f99" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01e78f21b22448058e71c375777316ae"
I20260812 18:53:41.172115 18380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6be00391fd343c5bcd343cef37a9f99" candidate_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7254da3822204c4290b6eb78f491ea38"
I20260812 18:53:41.172241 18054 raft_consensus.cc:3037] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:41.172255 18380 raft_consensus.cc:3037] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:41.173264 18054 raft_consensus.cc:2454] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 1.
I20260812 18:53:41.173269 18380 raft_consensus.cc:2454] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94e51d5ff9df422fae0c5b0068b2ad39 in term 1.
I20260812 18:53:41.173633 18138 leader_election.cc:302] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7254da3822204c4290b6eb78f491ea38, 94e51d5ff9df422fae0c5b0068b2ad39; no voters:
I20260812 18:53:41.173807 18495 raft_consensus.cc:2781] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:41.173902 18495 raft_consensus.cc:686] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 LEADER]: Becoming Leader. State: Replica: 94e51d5ff9df422fae0c5b0068b2ad39, State: Running, Role: LEADER
I20260812 18:53:41.174048 18495 consensus_queue.cc:237] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } }
I20260812 18:53:41.175340 18480 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: TxnStatusTablet state changed. Reason: New leader 94e51d5ff9df422fae0c5b0068b2ad39. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.175482 18480 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:41.175791 18520 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 18:53:41.176043 18520 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 18:53:41.176103 17876 catalog_manager.cc:5495] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } health_report { overall_health: UNKNOWN } } }
I20260812 18:53:41.205101 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.201652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.16.40.194:37689)
0812 18:53:41.201781 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.201788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.201805 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:53:41.202039 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.202043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.202107 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.202188 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:41.202475 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.202993 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.203920 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.204136 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.204250 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.204618 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.204696 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.204781 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.204885 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.204951 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 18:53:41.205101 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.201541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37689 (local address 127.0.0.1:46164)
0812 18:53:41.201738 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:41.201755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:41.201882 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:41.202224 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:41.202226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:41.202234 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:41.202360 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.202365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.203110 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.203115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.203732 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.203741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.204253 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.204256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.204368 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.204424 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.204785 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.204806 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 18:53:41.204873 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 18:53:41.204964 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 18:53:41.208742 18054 raft_consensus.cc:1261] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Refusing update from remote peer 94e51d5ff9df422fae0c5b0068b2ad39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:41.208742 18380 raft_consensus.cc:1261] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Refusing update from remote peer 94e51d5ff9df422fae0c5b0068b2ad39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:41.209244 18495 consensus_queue.cc:1035] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:41.209452 18480 consensus_queue.cc:1035] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:41.211007 18475 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: TxnStatusTablet state changed. Reason: New leader 94e51d5ff9df422fae0c5b0068b2ad39. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.211143 18475 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.211202 18478 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: TxnStatusTablet state changed. Reason: New leader 94e51d5ff9df422fae0c5b0068b2ad39. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.211337 18478 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.214041 18495 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.214190 18495 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:41.214299 18495 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.214372 18495 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 94e51d5ff9df422fae0c5b0068b2ad39: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:41.215694 18475 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.215845 18475 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 7254da3822204c4290b6eb78f491ea38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.216109 18478 tablet_replica.cc:439] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 } } peers { permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 } } peers { permanent_uuid: "7254da3822204c4290b6eb78f491ea38" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 43997 } } }
I20260812 18:53:41.216228 18478 tablet_replica.cc:442] T c6be00391fd343c5bcd343cef37a9f99 P 01e78f21b22448058e71c375777316ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:41.217098 18526 mvcc.cc:205] Tried to move back new op lower bound from 7317753123665645568 to 7317753123530694656. Current Snapshot: MvccSnapshot[applied={T|T < 7317753123665645568}]
I20260812 18:53:41.226059 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.221138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.16.40.195:43997)
0812 18:53:41.221357 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.221362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.221378 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:53:41.221505 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.221508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.221572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.221698 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:41.222045 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.222906 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.224089 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.224478 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.224642 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.224947 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.225572 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.225693 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.225794 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.225859 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260812 18:53:41.239142 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.234860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43757 (local address 127.16.40.194:37689)
0812 18:53:41.235016 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.235023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.235262 (+ 239us) server_negotiation.cc:407] Connection header received
0812 18:53:41.235436 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.235439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.235498 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.235579 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:41.235856 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.236368 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.237999 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.238229 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.238347 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.238665 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.238746 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.238829 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.238928 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.238985 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:53:41.239143 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.234769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37689 (local address 127.16.40.195:43757)
0812 18:53:41.235174 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:41.235187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:41.235317 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:41.235604 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:41.235606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:41.235615 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:41.235749 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.235754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.236486 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.236491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.237853 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.237865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.238352 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.238356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.238459 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.238511 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.238833 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.238843 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 18:53:41.238909 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 18:53:41.239002 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":110,"threads_started":1}
I20260812 18:53:41.247928 18054 raft_consensus.cc:1261] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Refusing update from remote peer 7254da3822204c4290b6eb78f491ea38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:41.248178 18214 raft_consensus.cc:1261] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Refusing update from remote peer 7254da3822204c4290b6eb78f491ea38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:41.248879 18475 consensus_queue.cc:1035] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94e51d5ff9df422fae0c5b0068b2ad39" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:41.249126 18506 consensus_queue.cc:1035] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01e78f21b22448058e71c375777316ae" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:41.254244 18524 mvcc.cc:205] Tried to move back new op lower bound from 7317753123826167808 to 7317753122767749120. Current Snapshot: MvccSnapshot[applied={T|T < 7317753123826167808}]
I20260812 18:53:41.273550 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.269729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:45319 (local address 127.16.40.195:43997)
0812 18:53:41.269869 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.269874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.269887 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:53:41.269934 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.269937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.269992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.270083 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:41.270331 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.271112 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.272272 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.272640 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.272736 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.273123 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.273200 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.273260 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.273380 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.273435 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:53:41.273600 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.269357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:43997 (local address 127.16.40.194:45319)
0812 18:53:41.269661 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:41.269680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:41.269788 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:41.270101 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:41.270103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:41.270111 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:41.270225 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.270230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.271247 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.271251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.272143 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:41.272155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.272820 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.272824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:41.272927 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.272983 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:41.273287 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:41.273294 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:53:41.273359 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 18:53:41.273444 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260812 18:53:41.313735 17877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:41.313943 17877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:41.316654 17877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307657da93d348b8b6e04c1db040b515: Sending DeleteTablet for 3 replicas of tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4
I20260812 18:53:41.317384 18032 tablet_service.cc:1514] Processing DeleteTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:41 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:42568
I20260812 18:53:41.317476 18193 tablet_service.cc:1514] Processing DeleteTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:41 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46136
I20260812 18:53:41.317804 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:41 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:59984
I20260812 18:53:41.318378 18549 tablet_replica.cc:330] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: stopping tablet replica
I20260812 18:53:41.318595 18549 raft_consensus.cc:2229] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:41.318795 18549 raft_consensus.cc:2258] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:41.319722 18549 ts_tablet_manager.cc:1905] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:41.323506 18552 tablet_replica.cc:330] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: stopping tablet replica
I20260812 18:53:41.323679 18552 raft_consensus.cc:2229] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:41.323559 18549 ts_tablet_manager.cc:1918] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:41.323887 18549 log.cc:1196] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/a94bf73e5fd24bd5b2c6b8973a8d1bb4
I20260812 18:53:41.324072 18552 raft_consensus.cc:2258] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:41.324456 18549 ts_tablet_manager.cc:1939] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 01e78f21b22448058e71c375777316ae: Deleting consensus metadata
I20260812 18:53:41.325426 17862 catalog_manager.cc:4841] TS 01e78f21b22448058e71c375777316ae (127.16.40.193:43237): tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (table test-table [id=5d6dbb69db444604b48910dd61474427]) successfully deleted
I20260812 18:53:41.326457 18552 ts_tablet_manager.cc:1905] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 18:53:41 dist-test-slave-rnrw krb5kdc[17829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:41.330137 18552 ts_tablet_manager.cc:1918] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:41.330264 18552 log.cc:1196] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/a94bf73e5fd24bd5b2c6b8973a8d1bb4
I20260812 18:53:41.330580 18552 ts_tablet_manager.cc:1939] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 7254da3822204c4290b6eb78f491ea38: Deleting consensus metadata
I20260812 18:53:41.331558 18551 tablet_replica.cc:330] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: stopping tablet replica
I20260812 18:53:41.331612 17861 catalog_manager.cc:4841] TS 7254da3822204c4290b6eb78f491ea38 (127.16.40.195:43997): tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (table test-table [id=5d6dbb69db444604b48910dd61474427]) successfully deleted
I20260812 18:53:41.331765 18551 raft_consensus.cc:2229] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:41.331947 18551 raft_consensus.cc:2258] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:41.333834 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.319246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.16.40.193:43237)
0812 18:53:41.319365 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.319370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.319934 (+ 564us) server_negotiation.cc:407] Connection header received
0812 18:53:41.320143 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.320147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.320202 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.320258 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:41.321298 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.322128 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.323589 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.323923 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.324005 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.329040 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:41.329059 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:41.329078 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:41.329118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:41.331401 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.332037 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.332045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.332047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.332160 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.332595 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.332600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.332601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.332989 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:41.333236 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.333476 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.333633 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 18:53:41.334666 18100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42570
I20260812 18:53:41.337229 18551 ts_tablet_manager.cc:1905] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:41.340646 18551 ts_tablet_manager.cc:1918] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:41.340754 18551 log.cc:1196] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/a94bf73e5fd24bd5b2c6b8973a8d1bb4
I20260812 18:53:41.341032 18551 ts_tablet_manager.cc:1939] T a94bf73e5fd24bd5b2c6b8973a8d1bb4 P 94e51d5ff9df422fae0c5b0068b2ad39: Deleting consensus metadata
I20260812 18:53:41.341909 17861 catalog_manager.cc:4841] TS 94e51d5ff9df422fae0c5b0068b2ad39 (127.16.40.194:37689): tablet a94bf73e5fd24bd5b2c6b8973a8d1bb4 (table test-table [id=5d6dbb69db444604b48910dd61474427]) successfully deleted
I20260812 18:53:41.348443 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:41.336821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.16.40.254:37587)
0812 18:53:41.336963 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:41.336968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:41.337920 (+ 952us) server_negotiation.cc:407] Connection header received
0812 18:53:41.338131 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:41.338135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:41.338190 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:41.338245 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:41.339241 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.340106 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.341737 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:41.342116 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:41.342211 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:41.344365 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:41.344369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:41.344372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:41.344408 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:41.346358 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.346931 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.346935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.346937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.347031 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:41.347327 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:41.347330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:41.347332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:41.347602 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:41.347709 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:41.348088 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 18:53:41.348246 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 18:53:41.349702 17877 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33866
I20260812 18:53:41.351490 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17957
I20260812 18:53:41.374554 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 18116
I20260812 18:53:41.389212 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 18277
I20260812 18:53:41.400856 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 17842
2026-08-12T18:53:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3185 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:41 dist-test-slave-rnrw krb5kdc[18559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:41 dist-test-slave-rnrw krb5kdc[18559](info): set up 2 sockets
Aug 12 18:53:41 dist-test-slave-rnrw krb5kdc[18559](info): commencing operation
krb5kdc: starting...
W20260812 18:53:42.555948 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:42 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:42.696182 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39911
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:39911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:42.818107 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:42.818445 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:42.818522 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:42.823246 18573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:42.823362 18573 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 18:53:42.823396 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:42.823439 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:42.823477 18573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:42.823513 18573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:42.829375 18573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:39911
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:42.830650 18573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:42.831775 18573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:42.840987 18579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:42.840987 18578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:42.841140 18573 server_base.cc:1034] running on GCE node
W20260812 18:53:42.840987 18581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:42.841893 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:42.843099 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:42.844338 18573 hybrid_clock.cc:648] HybridClock initialized: now 1786560822844322 us; error 30 us; skew 500 ppm
Aug 12 18:53:42 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560822, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:42.847675 18573 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:42.848958 18573 webserver.cc:460] Webserver started at http://127.16.40.254:36931/ using document root <none> and password file <none>
I20260812 18:53:42.849598 18573 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:42.849702 18573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:42.849958 18573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:42.852334 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "9e9af9e31579499a8a50cf95d2b08c8d"
format_stamp: "Formatted at 2026-08-12 18:53:42 on dist-test-slave-rnrw"
server_key: "0d6356628e39ca6a709833a6373aadf4"
server_key_iv: "162381ef0fa0d31bab0badcbe431a7f6"
server_key_version: "encryptionkey@0"
I20260812 18:53:42.853104 18573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "9e9af9e31579499a8a50cf95d2b08c8d"
format_stamp: "Formatted at 2026-08-12 18:53:42 on dist-test-slave-rnrw"
server_key: "0d6356628e39ca6a709833a6373aadf4"
server_key_iv: "162381ef0fa0d31bab0badcbe431a7f6"
server_key_version: "encryptionkey@0"
I20260812 18:53:42.857594 18573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:42.860335 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:42.861879 18573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:53:42.862053 18573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "9e9af9e31579499a8a50cf95d2b08c8d"
format_stamp: "Formatted at 2026-08-12 18:53:42 on dist-test-slave-rnrw"
server_key: "0d6356628e39ca6a709833a6373aadf4"
server_key_iv: "162381ef0fa0d31bab0badcbe431a7f6"
server_key_version: "encryptionkey@0"
I20260812 18:53:42.862213 18573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:42.876886 18573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:42.880131 18573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:42.880380 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:42.889514 18573 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:39911
I20260812 18:53:42.889536 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:39911 every 8 connection(s)
I20260812 18:53:42.890594 18573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:42.893745 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:42.894618 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 18573
I20260812 18:53:42.894721 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:42.895006 16547 external_mini_cluster.cc:1428] Setting key 27497c48a413e0405ab2198c1d1087de
I20260812 18:53:42.901206 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Bootstrap starting.
I20260812 18:53:42.904407 18664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Neither blocks nor log segments found. Creating new log.
Aug 12 18:53:42 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:42.905332 18664 log.cc:824] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:42.907691 18664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: No bootstrap required, opened a new log
I20260812 18:53:42.911135 18664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } }
I20260812 18:53:42.911464 18664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:42.911550 18664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9af9e31579499a8a50cf95d2b08c8d, State: Initialized, Role: FOLLOWER
I20260812 18:53:42.911890 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:42.896458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.16.40.254:39911)
0812 18:53:42.896997 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:42.897011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:42.897049 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:53:42.897701 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:42.897720 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:42.898069 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:42.898367 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:42.899388 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:42.900415 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:42.901694 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:42.902217 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:42.902351 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:42.905555 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:42.905561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:42.905579 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:42.905612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:42.908370 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:42.909036 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:42.909042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:42.909048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:42.909197 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:42.909542 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:42.909545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:42.909546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:42.910008 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:42.910267 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:42.910563 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 18:53:42.910914 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":157,"threads_started":1}
I20260812 18:53:42.912084 18664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } }
I20260812 18:53:42.912258 18664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:42.912350 18664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:42.912458 18664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:42.913767 18664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } }
I20260812 18:53:42.914147 18664 leader_election.cc:302] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e9af9e31579499a8a50cf95d2b08c8d; no voters:
I20260812 18:53:42.914543 18664 leader_election.cc:288] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:42.914625 18671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:42.914842 18671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [term 1 LEADER]: Becoming Leader. State: Replica: 9e9af9e31579499a8a50cf95d2b08c8d, State: Running, Role: LEADER
I20260812 18:53:42.915182 18671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } }
I20260812 18:53:42.915952 18664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:42.917182 18673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e9af9e31579499a8a50cf95d2b08c8d. Latest consensus state: current_term: 1 leader_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } } }
I20260812 18:53:42.917205 18672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e9af9e31579499a8a50cf95d2b08c8d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39911 } } }
I20260812 18:53:42.917454 18672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:42.917447 18673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:42.918648 18680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:42.921746 18680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:42.927873 18680 catalog_manager.cc:1347] Generated new cluster ID: 7e8fada693df4dd78c49930771c69e0a
I20260812 18:53:42.927969 18680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:42.950191 18680 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:42.951189 18680 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:42.957751 18680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Generated new TSK 0
I20260812 18:53:42.958534 18680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:42.992488 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:43.116143 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:43.116468 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:43.116521 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:43.121119 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:43.121208 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:43.121253 18694 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 18:53:43.121292 18694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:43.121361 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:43.127185 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:39911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:43.128513 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:43.129796 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:43.140039 18700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:43.140097 18701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:43.140221 18694 server_base.cc:1034] running on GCE node
W20260812 18:53:43.140039 18703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:43.140852 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:43.141639 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:43.142839 18694 hybrid_clock.cc:648] HybridClock initialized: now 1786560823142804 us; error 48 us; skew 500 ppm
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:43.145895 18694 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:43.146881 18694 webserver.cc:460] Webserver started at http://127.16.40.193:41277/ using document root <none> and password file <none>
I20260812 18:53:43.147536 18694 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:43.147637 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:43.147895 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:43.150197 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "026cb669b5ca451eba23e0078cafcb5f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "361a5a8905797885ebae165179c957c8"
server_key_iv: "1dda33a70171076b8d02d2fe371f7963"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.150820 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "026cb669b5ca451eba23e0078cafcb5f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "361a5a8905797885ebae165179c957c8"
server_key_iv: "1dda33a70171076b8d02d2fe371f7963"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.154994 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:43.157727 18712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.159368 18694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:43.159552 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "026cb669b5ca451eba23e0078cafcb5f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "361a5a8905797885ebae165179c957c8"
server_key_iv: "1dda33a70171076b8d02d2fe371f7963"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.159696 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:43.189458 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:43.192977 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:43.193214 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:43.194821 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:43.194901 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.195000 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:43.195046 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:43.206802 18694 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:34099
I20260812 18:53:43.206882 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:34099 every 8 connection(s)
I20260812 18:53:43.208123 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:43.210184 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.195800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:33577 (local address 127.16.40.254:39911)
0812 18:53:43.195936 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.195939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.197092 (+ 1153us) server_negotiation.cc:407] Connection header received
0812 18:53:43.198329 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.198332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.198400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.198498 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.200271 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.201102 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.202559 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.202818 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.202943 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.205902 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.205906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.205909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.205941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.207824 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.208418 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.208422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.208427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.208536 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.209007 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.209011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.209012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.209271 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.209357 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.209921 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.210042 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:53:43.211006 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.196381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.193:33577)
0812 18:53:43.196923 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.196969 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.198002 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.198743 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.198755 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.199279 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.200066 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.200090 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.201296 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.201299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.202420 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.202431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.203007 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.203012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.203212 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.203883 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.203901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.205698 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.207947 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.207952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.207965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.208312 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.208701 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.208706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.208708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.208880 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.209379 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.209384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.209694 (+ 310us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.209861 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.210134 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":185,"threads_started":1}
I20260812 18:53:43.211285 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 18694
I20260812 18:53:43.211413 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:43.211763 16547 external_mini_cluster.cc:1428] Setting key 1c3070a32f5352afc1843c7b53e37de2
I20260812 18:53:43.212783 18843 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39911
I20260812 18:53:43.212937 18843 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:43.213753 18843 heartbeater.cc:511] Master 127.16.40.254:39911 requested a full tablet report, sending...
I20260812 18:53:43.215657 18605 ts_manager.cc:194] Registered new tserver with Master: 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099)
I20260812 18:53:43.217092 18605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:33577
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:43.225435 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.215261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51301 (local address 127.16.40.254:39911)
0812 18:53:43.215392 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.215396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.217210 (+ 1814us) server_negotiation.cc:407] Connection header received
0812 18:53:43.217360 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.217363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.217409 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.217499 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.218099 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.218685 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.220008 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.220248 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.220396 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.221773 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.221777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.221780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.221805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.223373 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.223932 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.223936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.223938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.224041 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.224479 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.224483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.224485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.224740 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.224857 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.225129 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.225258 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:53:43.225443 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.215207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.193:51301)
0812 18:53:43.217109 (+ 1902us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.217129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.217257 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.217524 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.217530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.217793 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.217962 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.217974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.218822 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.218829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.219848 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.219860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.220405 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.220409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.220649 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.221237 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.221250 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.221621 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.223502 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.223507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.223554 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.223797 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.224191 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.224197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.224204 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.224372 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.224845 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.224848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.224953 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.225072 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.225237 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":30,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:43.246093 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:43.370103 18854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:43.370445 18854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:43.370536 18854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:43.374941 18854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:43.375047 18854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:43.375128 18854 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 18:53:43.375171 18854 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:43.375242 18854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:43.381134 18854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:39911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:43.382450 18854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:43.383728 18854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:43.394573 18862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:43.394573 18859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:43.394579 18860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:43.395069 18854 server_base.cc:1034] running on GCE node
I20260812 18:53:43.395546 18854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:43.396342 18854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:43.397578 18854 hybrid_clock.cc:648] HybridClock initialized: now 1786560823397564 us; error 37 us; skew 500 ppm
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:43.401113 18854 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:43.402271 18854 webserver.cc:460] Webserver started at http://127.16.40.194:33121/ using document root <none> and password file <none>
I20260812 18:53:43.402949 18854 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:43.403079 18854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:43.403354 18854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:43.405855 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "95a892233422466abdb0b567bcd5bb7f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "b3c8f229cfe488b2db08774dd2744ab8"
server_key_iv: "e782a3f5f1c06da78b1f8bcc81ebe5d5"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.406533 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "95a892233422466abdb0b567bcd5bb7f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "b3c8f229cfe488b2db08774dd2744ab8"
server_key_iv: "e782a3f5f1c06da78b1f8bcc81ebe5d5"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.411957 18854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 18:53:43.415056 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.416772 18854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:43.416970 18854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "95a892233422466abdb0b567bcd5bb7f"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "b3c8f229cfe488b2db08774dd2744ab8"
server_key_iv: "e782a3f5f1c06da78b1f8bcc81ebe5d5"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.417135 18854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:43.437225 18854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:43.441044 18854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:43.441293 18854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:43.442898 18854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:43.442953 18854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.442997 18854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:43.443054 18854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:43.455919 18854 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:42957
I20260812 18:53:43.455987 19000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:42957 every 8 connection(s)
I20260812 18:53:43.457156 18854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:43.458521 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.443976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53889 (local address 127.16.40.254:39911)
0812 18:53:43.444150 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.444157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.445174 (+ 1017us) server_negotiation.cc:407] Connection header received
0812 18:53:43.446503 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.446508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.446577 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.446681 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.448624 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.449270 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.450677 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.450914 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.451047 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.454072 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.454076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.454078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.454106 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.455864 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.456615 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.456623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.456625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.456710 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.457114 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.457119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.457121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.457399 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.457542 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.458179 (+ 637us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.458326 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:53:43.459405 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.444380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.194:53889)
0812 18:53:43.444968 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.445024 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.446189 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.446842 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.446849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.447498 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.448327 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.448362 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.449453 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.449457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.450522 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.450534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.451115 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.451119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.451336 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.452074 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.452096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.453861 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.456007 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.456021 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.456038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.456407 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.456814 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.456818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.456820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.456979 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.457531 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.457538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.457895 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.458130 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.458471 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":243,"threads_started":1}
I20260812 18:53:43.460929 19001 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39911
I20260812 18:53:43.461133 19001 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:43.461686 19001 heartbeater.cc:511] Master 127.16.40.254:39911 requested a full tablet report, sending...
I20260812 18:53:43.462864 18611 ts_manager.cc:194] Registered new tserver with Master: 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:43.463783 18611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:53889
I20260812 18:53:43.465097 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 18854
I20260812 18:53:43.465229 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:43.465493 16547 external_mini_cluster.cc:1428] Setting key 99e2d803e5cea298f1225d67f85e6092
I20260812 18:53:43.472256 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.463390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.194:43519)
0812 18:53:43.463709 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.463725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.463843 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.464241 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.464243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.464532 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.464705 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.464713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.465760 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.465764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.466612 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.466654 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.467274 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.467278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.467355 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.468015 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.468031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.468374 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.470381 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.470388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.470391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.470704 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.471064 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.471067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.471068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.471155 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.471733 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.471735 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.471834 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.471934 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.472081 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":127,"threads_started":1}
I20260812 18:53:43.472425 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.463444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:43519 (local address 127.16.40.254:39911)
0812 18:53:43.463594 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.463599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.463784 (+ 185us) server_negotiation.cc:407] Connection header received
0812 18:53:43.463950 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.463953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.464002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.464099 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.464834 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.465606 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.466776 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.467114 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.467252 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.468528 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.468534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.468570 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.468604 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.470257 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.470840 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.470846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.470851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.470952 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.471254 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.471260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.471265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.471615 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.471731 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.472017 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.472178 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:43.501539 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:43.634719 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:43.635062 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:43.635152 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:43.640094 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:43.640190 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:43.640266 19013 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 18:53:43.640313 19013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:53:43.640393 19013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:43.646384 19013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:39911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:43.647598 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:43.648743 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:43.659163 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:43.659183 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:43.659278 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:43.659435 19013 server_base.cc:1034] running on GCE node
I20260812 18:53:43.660179 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:43.661088 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:43.662321 19013 hybrid_clock.cc:648] HybridClock initialized: now 1786560823662292 us; error 51 us; skew 500 ppm
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:43.665696 19013 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:43.666755 19013 webserver.cc:460] Webserver started at http://127.16.40.195:33645/ using document root <none> and password file <none>
I20260812 18:53:43.667434 19013 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:43.667501 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:43.667676 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:43.669873 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "7c28ee6af69dceaafe56c8795d80c20f"
server_key_iv: "90a86f94dbf72e1d542ff199c2292e1a"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.670403 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "7c28ee6af69dceaafe56c8795d80c20f"
server_key_iv: "90a86f94dbf72e1d542ff199c2292e1a"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.674396 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:43.676985 19030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.678268 19013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 18:53:43.678408 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0"
format_stamp: "Formatted at 2026-08-12 18:53:43 on dist-test-slave-rnrw"
server_key: "7c28ee6af69dceaafe56c8795d80c20f"
server_key_iv: "90a86f94dbf72e1d542ff199c2292e1a"
server_key_version: "encryptionkey@0"
I20260812 18:53:43.678534 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:43.691846 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:43.695506 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:43.695744 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:43.697458 19013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:43.697547 19013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.697629 19013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:43.697680 19013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:43.709322 19013 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:34009
I20260812 18:53:43.709502 19164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:34009 every 8 connection(s)
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:43.710502 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:43.715166 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.698466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:33455 (local address 127.16.40.254:39911)
0812 18:53:43.698616 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.698620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.700032 (+ 1412us) server_negotiation.cc:407] Connection header received
0812 18:53:43.701515 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.701523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.701608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.701728 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.703889 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.704913 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.706915 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.707349 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.707509 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.710685 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.710689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.710691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.710731 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.712973 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.713590 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.713595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.713597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.713698 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.714061 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.714064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.714065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.714283 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.714369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.714902 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.715026 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:53:43.716138 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.699058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.195:33455)
0812 18:53:43.699831 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.699886 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.701163 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.702011 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.702023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.702712 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.703666 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.703692 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.705088 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.705091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.706712 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.706728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.707607 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.707614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.707913 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.708732 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.708751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.710477 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.713111 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.713117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.713130 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.713432 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.713801 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.713804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.713805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.713962 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.714382 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.714388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.714660 (+ 272us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.714832 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.715133 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":152,"threads_started":1}
I20260812 18:53:43.717772 19165 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39911
I20260812 18:53:43.718053 19165 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:43.718765 19165 heartbeater.cc:511] Master 127.16.40.254:39911 requested a full tablet report, sending...
I20260812 18:53:43.720069 18605 ts_manager.cc:194] Registered new tserver with Master: ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009)
I20260812 18:53:43.720362 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19013
I20260812 18:53:43.720485 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:43.720866 18605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:33455
I20260812 18:53:43.720927 16547 external_mini_cluster.cc:1428] Setting key 5602c440dcb7e480d47ce25377aae825
I20260812 18:53:43.723143 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:53:43.731010 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.720663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:40471 (local address 127.16.40.254:39911)
0812 18:53:43.720823 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.720827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.721550 (+ 723us) server_negotiation.cc:407] Connection header received
0812 18:53:43.721677 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.721680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.721731 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.721817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.722480 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.724857 (+ 2377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.725736 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.725963 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.726131 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.727350 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.727354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.727356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.727380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.729121 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.729687 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.729690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.729692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.729779 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.730112 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.730114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.730115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.730313 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.730396 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.730763 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.730879 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 18:53:43.731307 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.720570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.16.40.195:40471)
0812 18:53:43.721445 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.721461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.721574 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.721938 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.721941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.722187 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.722356 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.722366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.725013 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.725016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.725586 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.725595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.726148 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.726151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.726252 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.726828 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.726840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.727199 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.729253 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.729257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.729260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.729569 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.729890 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.729896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.729901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.730011 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.730410 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.730415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.730560 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.730684 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.730879 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":9,"thread_start_us":174,"threads_started":1}
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:43.752791 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.744379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.16.40.254:39911)
0812 18:53:43.744529 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.744534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.744585 (+ 51us) server_negotiation.cc:407] Connection header received
0812 18:53:43.744645 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.744647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.744691 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.744772 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.745509 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.746061 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.746956 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.747177 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.747271 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.749903 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.749907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.749909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.749936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.751273 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.751662 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.751665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.751666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.751742 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.751979 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.751981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.751982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.752178 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.752280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.752485 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.752663 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:53:43.756049 18605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:43.759379 18605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:43.775805 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.769071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34099 (local address 127.0.0.1:57118)
0812 18:53:43.769650 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.769702 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.769841 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.771214 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.771223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.771256 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:43.771454 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.771462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.772914 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.772917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.774324 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.774334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.775216 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.775220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.775339 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.775381 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.775506 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.775644 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":55,"thread_start_us":103,"threads_started":1}
I20260812 18:53:43.775805 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.769208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.16.40.193:34099)
0812 18:53:43.770671 (+ 1463us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.770680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.770708 (+ 28us) server_negotiation.cc:407] Connection header received
0812 18:53:43.770790 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.770795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.771021 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.771194 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:43.771631 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.772750 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.774463 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.775076 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.775215 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.775387 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.775551 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.775656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":60,"threads_started":1}
I20260812 18:53:43.777274 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.769559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42957 (local address 127.0.0.1:54494)
0812 18:53:43.770153 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.770166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.770264 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.772302 (+ 2038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.772309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.772335 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:43.772477 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.772484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.773972 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.773976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.775253 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.775264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.776484 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.776494 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.776908 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.776932 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.777013 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.777093 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":5,"thread_start_us":113,"threads_started":1}
I20260812 18:53:43.777748 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.770163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.16.40.194:42957)
0812 18:53:43.770670 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.770683 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.770708 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:53:43.770790 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.770795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.771021 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.772095 (+ 1074us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:43.772654 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.773828 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.775407 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.776063 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.776220 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.776802 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.777518 (+ 716us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.777635 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":153,"threads_started":1}
I20260812 18:53:43.779258 18767 tablet_service.cc:1467] Processing CreateTablet for tablet dd5697b472cf41bba352c506ef30effb (DEFAULT_TABLE table=test-table [id=4fcc0944a9ca487eb5060b150483987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:43.780138 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.771631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34009 (local address 127.0.0.1:46608)
0812 18:53:43.774268 (+ 2637us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.774284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.774376 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.776308 (+ 1932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.776312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.776331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:43.776430 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.776435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.777844 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.777849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.778700 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.778711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.779765 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.779768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.779836 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.779889 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.779959 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.780029 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2566,"thread_start_us":107,"threads_started":1}
I20260812 18:53:43.780251 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.772097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.16.40.195:34009)
0812 18:53:43.774683 (+ 2586us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.774696 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.774722 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:53:43.774800 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.774806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.775035 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.775193 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:43.776606 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.777705 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.778850 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.779632 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.779755 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.779924 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.779985 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.780087 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2447,"thread_start_us":105,"threads_started":1}
I20260812 18:53:43.780864 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5697b472cf41bba352c506ef30effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:43.783447 19086 tablet_service.cc:1467] Processing CreateTablet for tablet dd5697b472cf41bba352c506ef30effb (DEFAULT_TABLE table=test-table [id=4fcc0944a9ca487eb5060b150483987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:43.783663 18927 tablet_service.cc:1467] Processing CreateTablet for tablet dd5697b472cf41bba352c506ef30effb (DEFAULT_TABLE table=test-table [id=4fcc0944a9ca487eb5060b150483987b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:43.784945 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5697b472cf41bba352c506ef30effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:43.784970 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5697b472cf41bba352c506ef30effb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:43.789659 19191 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Bootstrap starting.
I20260812 18:53:43.792398 19191 tablet_bootstrap.cc:654] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:43.793459 19191 log.cc:824] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:43.793581 19193 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Bootstrap starting.
I20260812 18:53:43.794701 19192 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Bootstrap starting.
I20260812 18:53:43.795804 19191 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: No bootstrap required, opened a new log
I20260812 18:53:43.796057 19191 ts_tablet_manager.cc:1397] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 18:53:43.796051 19193 tablet_bootstrap.cc:654] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:43.797082 19192 tablet_bootstrap.cc:654] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:43.797062 19193 log.cc:824] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:43.797940 19192 log.cc:824] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:43.799466 19193 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: No bootstrap required, opened a new log
I20260812 18:53:43.799746 19193 ts_tablet_manager.cc:1397] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:53:43.800175 19192 tablet_bootstrap.cc:492] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: No bootstrap required, opened a new log
I20260812 18:53:43.800431 19192 ts_tablet_manager.cc:1397] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:53:43.800772 19191 raft_consensus.cc:348] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.801039 19191 raft_consensus.cc:374] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:43.801123 19191 raft_consensus.cc:729] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Initialized, Role: FOLLOWER
I20260812 18:53:43.801636 19191 consensus_queue.cc:260] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.802343 18843 heartbeater.cc:503] Master 127.16.40.254:39911 was elected leader, sending a full tablet report...
I20260812 18:53:43.802745 19191 ts_tablet_manager.cc:1428] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 18:53:43.804030 19193 raft_consensus.cc:348] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.804324 19193 raft_consensus.cc:374] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:43.804431 19193 raft_consensus.cc:729] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Initialized, Role: FOLLOWER
I20260812 18:53:43.804587 19192 raft_consensus.cc:348] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.804845 19192 raft_consensus.cc:374] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:43.804939 19192 raft_consensus.cc:729] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a892233422466abdb0b567bcd5bb7f, State: Initialized, Role: FOLLOWER
I20260812 18:53:43.804934 19193 consensus_queue.cc:260] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.805392 19192 consensus_queue.cc:260] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.805725 19165 heartbeater.cc:503] Master 127.16.40.254:39911 was elected leader, sending a full tablet report...
I20260812 18:53:43.806039 19193 ts_tablet_manager.cc:1428] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:53:43.807374 19001 heartbeater.cc:503] Master 127.16.40.254:39911 was elected leader, sending a full tablet report...
I20260812 18:53:43.807828 19192 ts_tablet_manager.cc:1428] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:53:43.914139 19198 raft_consensus.cc:482] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:43.914426 19198 raft_consensus.cc:504] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.915735 19198 leader_election.cc:288] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:43.920645 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.916633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:47413 (local address 127.16.40.194:42957)
0812 18:53:43.916889 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.916896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.917066 (+ 170us) server_negotiation.cc:407] Connection header received
0812 18:53:43.917178 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.917182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.917235 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.917340 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:43.917695 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.918329 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.919621 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.920022 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.920158 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.920218 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.920415 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.920492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 18:53:43.920643 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.916290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:47229 (local address 127.16.40.195:34009)
0812 18:53:43.916431 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.916436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.917144 (+ 708us) server_negotiation.cc:407] Connection header received
0812 18:53:43.917419 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.917422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.917502 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.917600 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:43.917906 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.918514 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.919606 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.920011 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.920154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.920214 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.920399 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.920475 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:53:43.920761 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34009 (local address 127.16.40.193:47229)
0812 18:53:43.917043 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.917059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.917231 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.917612 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.917615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.917633 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:43.917768 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.917773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.918644 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.918648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.919477 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.919487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.920142 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.920146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.920248 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.920273 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.920373 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.920456 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":544,"thread_start_us":136,"threads_started":1}
I20260812 18:53:43.920768 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42957 (local address 127.16.40.193:47413)
0812 18:53:43.916472 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.916492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.916841 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.917399 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.917402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.917423 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:43.917571 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.917576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.918504 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.918507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.919485 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.919495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.920151 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.920155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.920284 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.920303 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.920396 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.920488 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":15,"thread_start_us":142,"threads_started":1}
I20260812 18:53:43.921506 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5697b472cf41bba352c506ef30effb" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" is_pre_election: true
I20260812 18:53:43.921627 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5697b472cf41bba352c506ef30effb" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f" is_pre_election: true
I20260812 18:53:43.921908 18952 raft_consensus.cc:2454] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 0.
I20260812 18:53:43.921908 19115 raft_consensus.cc:2454] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 0.
I20260812 18:53:43.922538 18716 leader_election.cc:302] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:43.922892 19198 raft_consensus.cc:2781] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:43.922993 19198 raft_consensus.cc:482] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:43.923027 19198 raft_consensus.cc:3037] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:43.924201 19198 raft_consensus.cc:504] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.924679 19198 leader_election.cc:288] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 election: Requested vote from peers ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:43.925041 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5697b472cf41bba352c506ef30effb" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0"
I20260812 18:53:43.925041 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5697b472cf41bba352c506ef30effb" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f"
I20260812 18:53:43.925185 19115 raft_consensus.cc:3037] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:43.925185 18952 raft_consensus.cc:3037] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:43.926558 18952 raft_consensus.cc:2454] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 1.
I20260812 18:53:43.926558 19115 raft_consensus.cc:2454] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 1.
I20260812 18:53:43.926909 18713 leader_election.cc:302] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, 95a892233422466abdb0b567bcd5bb7f; no voters:
I20260812 18:53:43.927106 19198 raft_consensus.cc:2781] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:43.927423 19198 raft_consensus.cc:686] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 1 LEADER]: Becoming Leader. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Running, Role: LEADER
I20260812 18:53:43.927840 19198 consensus_queue.cc:237] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } }
I20260812 18:53:43.931238 18611 catalog_manager.cc:5495] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193). New cstate: current_term: 1 leader_uuid: "026cb669b5ca451eba23e0078cafcb5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } health_report { overall_health: UNKNOWN } } }
W20260812 18:53:43.958561 19002 tablet.cc:2369] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:53:43.959932 18845 tablet.cc:2369] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:53:43.962580 19166 tablet.cc:2369] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 18:53:43 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560822, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:43.995944 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.986756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.16.40.254:39911)
0812 18:53:43.986967 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:43.986972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:43.987188 (+ 216us) server_negotiation.cc:407] Connection header received
0812 18:53:43.987357 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:43.987360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:43.987416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:43.987508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:43.988264 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.988865 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.989675 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.989889 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.990001 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.992522 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:43.992526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:43.992528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:43.992614 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:43.994174 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.994739 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.994741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.994743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.994819 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:43.995076 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:43.995078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:43.995079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:43.995283 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:43.995362 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:43.995689 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 18:53:43.995804 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:53:43.995949 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:43.986703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39911 (local address 127.0.0.1:56044)
0812 18:53:43.987088 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:43.987101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:43.987256 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:43.987577 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:43.987581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:43.988026 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:43.988135 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.988143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.988985 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.988990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.989546 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:43.989555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:43.990007 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:43.990011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:43.990106 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:43.990700 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:43.990714 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:43.992344 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:43.994310 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.994322 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.994328 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.994641 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.994903 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:43.994905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:43.994906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:43.994991 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:43.995378 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:43.995383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:43.995505 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 18:53:43.995617 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 18:53:43.995751 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
I20260812 18:53:44.000497 18611 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:56044:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 18:53:44.001528 18611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:44.006973 19086 tablet_service.cc:1467] Processing CreateTablet for tablet 71d8cbc17215441bb4e64793c5e65be8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b007318fcff4902b3ac8dcd3fdadb7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:44.006776 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 71d8cbc17215441bb4e64793c5e65be8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b007318fcff4902b3ac8dcd3fdadb7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:44.006883 18927 tablet_service.cc:1467] Processing CreateTablet for tablet 71d8cbc17215441bb4e64793c5e65be8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b007318fcff4902b3ac8dcd3fdadb7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:53:44.007637 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8cbc17215441bb4e64793c5e65be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.007637 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8cbc17215441bb4e64793c5e65be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.007696 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d8cbc17215441bb4e64793c5e65be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.010730 19192 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: Bootstrap starting.
I20260812 18:53:44.010733 19191 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: Bootstrap starting.
I20260812 18:53:44.010733 19193 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Bootstrap starting.
I20260812 18:53:44.011731 19192 tablet_bootstrap.cc:654] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.011878 19193 tablet_bootstrap.cc:654] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.012244 19191 tablet_bootstrap.cc:654] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.013370 19192 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: No bootstrap required, opened a new log
I20260812 18:53:44.013481 19192 ts_tablet_manager.cc:1397] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:44.013736 19191 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: No bootstrap required, opened a new log
I20260812 18:53:44.013823 19191 ts_tablet_manager.cc:1397] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:44.014056 19193 tablet_bootstrap.cc:492] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: No bootstrap required, opened a new log
I20260812 18:53:44.014029 19192 raft_consensus.cc:348] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.014135 19193 ts_tablet_manager.cc:1397] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:44.014145 19192 raft_consensus.cc:374] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.014180 19192 raft_consensus.cc:729] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a892233422466abdb0b567bcd5bb7f, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.014312 19192 consensus_queue.cc:260] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.014358 19191 raft_consensus.cc:348] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.014461 19191 raft_consensus.cc:374] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.014492 19191 raft_consensus.cc:729] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.014623 19192 ts_tablet_manager.cc:1428] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:44.014596 19193 raft_consensus.cc:348] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.014652 19191 consensus_queue.cc:260] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.014711 19193 raft_consensus.cc:374] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.014755 19193 raft_consensus.cc:729] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.014899 19191 ts_tablet_manager.cc:1428] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:44.014887 19193 consensus_queue.cc:260] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.015048 19200 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.015157 19193 ts_tablet_manager.cc:1428] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:44.015199 19200 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.015146 19198 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.015272 19198 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.015408 19199 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.015554 19199 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.072799 19199 raft_consensus.cc:482] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:44.072980 19199 raft_consensus.cc:504] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.074198 19199 leader_election.cc:288] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:44.078156 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.074501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42957 (local address 127.16.40.195:36341)
0812 18:53:44.074724 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:44.074742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:44.074842 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:44.075166 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:44.075168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:44.075187 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:44.075316 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.075322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.076100 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.076105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.077005 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.077022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.077686 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.077805 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.077824 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 18:53:44.077916 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 18:53:44.077993 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260812 18:53:44.078156 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.074771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:36341 (local address 127.16.40.194:42957)
0812 18:53:44.074930 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.074936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.074958 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:53:44.075009 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.075012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.075052 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.075150 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:44.075440 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.075960 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077151 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.077555 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077688 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.077745 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.077938 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.078008 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:53:44.078390 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.074666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:56377 (local address 127.16.40.193:34099)
0812 18:53:44.074811 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.074817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.075117 (+ 300us) server_negotiation.cc:407] Connection header received
0812 18:53:44.075277 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.075280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.075332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.075422 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:44.075729 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.076253 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077411 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.077809 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077969 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.078023 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.078209 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.078281 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:53:44.078418 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.074502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34099 (local address 127.16.40.195:56377)
0812 18:53:44.075005 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:44.075016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:44.075149 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:44.075435 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:44.075437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:44.075454 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:44.075595 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.075601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.076378 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.076383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.077270 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.077281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.077966 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.077970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.078075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.078095 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 18:53:44.078187 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 18:53:44.078259 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":22,"thread_start_us":131,"threads_started":1}
I20260812 18:53:44.078795 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8cbc17215441bb4e64793c5e65be8" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f" is_pre_election: true
I20260812 18:53:44.078974 18952 raft_consensus.cc:2454] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:44.079200 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8cbc17215441bb4e64793c5e65be8" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f" is_pre_election: true
I20260812 18:53:44.079393 19031 leader_election.cc:302] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:44.079478 18790 raft_consensus.cc:2454] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:44.079739 19199 raft_consensus.cc:2781] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:44.079823 19199 raft_consensus.cc:482] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:44.079874 19199 raft_consensus.cc:3037] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.080998 19199 raft_consensus.cc:504] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.081416 19199 leader_election.cc:288] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Requested vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:44.081848 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8cbc17215441bb4e64793c5e65be8" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f"
I20260812 18:53:44.081841 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d8cbc17215441bb4e64793c5e65be8" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f"
I20260812 18:53:44.081992 18790 raft_consensus.cc:3037] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.082005 18952 raft_consensus.cc:3037] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.083103 18790 raft_consensus.cc:2454] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:44.083124 18952 raft_consensus.cc:2454] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:44.083473 19032 leader_election.cc:302] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:44.083680 19199 raft_consensus.cc:2781] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:44.083932 19199 raft_consensus.cc:686] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 LEADER]: Becoming Leader. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Running, Role: LEADER
I20260812 18:53:44.084298 19199 consensus_queue.cc:237] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.086949 19218 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: TxnStatusTablet state changed. Reason: New leader ddc4ab0bc05c4a52991a3cf8aba3f2a0. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.087116 19218 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:44.087469 19220 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 18:53:44.087708 19220 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 18:53:44.088207 18605 catalog_manager.cc:5495] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } health_report { overall_health: HEALTHY } } }
I20260812 18:53:44.117477 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.112821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34009 (local address 127.0.0.1:46620)
0812 18:53:44.113001 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:44.113019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:44.113160 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:44.113556 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:44.113558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:44.113565 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:44.113724 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.113730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.114773 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.114776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.115382 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.115390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.116028 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.116032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.116130 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.116191 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.117121 (+ 930us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.117143 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 18:53:44.117223 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 18:53:44.117343 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 18:53:44.117522 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.112923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.16.40.195:34009)
0812 18:53:44.113062 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.113066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.113084 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:53:44.113340 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.113344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.113416 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.113537 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:44.113839 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.114652 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.115528 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.115874 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.116002 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.116352 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.116997 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.117116 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.117282 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.117347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:53:44.123970 18952 raft_consensus.cc:1261] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.123986 18790 raft_consensus.cc:1261] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.124747 19199 consensus_queue.cc:1035] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:44.124981 19218 consensus_queue.cc:1035] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:44.128671 19200 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: TxnStatusTablet state changed. Reason: New leader ddc4ab0bc05c4a52991a3cf8aba3f2a0. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.128829 19200 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.129083 19198 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: TxnStatusTablet state changed. Reason: New leader ddc4ab0bc05c4a52991a3cf8aba3f2a0. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.129211 19198 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.130854 19224 mvcc.cc:205] Tried to move back new op lower bound from 7317753135603855360 to 7317753135452569600. Current Snapshot: MvccSnapshot[applied={T|T < 7317753135603855360}]
I20260812 18:53:44.131369 19218 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.131474 19218 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:44.131770 19198 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.131878 19198 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 026cb669b5ca451eba23e0078cafcb5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.133097 19199 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.133208 19199 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:53:44.135461 19226 mvcc.cc:205] Tried to move back new op lower bound from 7317753135603855360 to 7317753135452569600. Current Snapshot: MvccSnapshot[applied={T|T < 7317753135603855360}]
I20260812 18:53:44.136036 19200 tablet_replica.cc:439] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } } }
I20260812 18:53:44.136175 19200 tablet_replica.cc:442] T 71d8cbc17215441bb4e64793c5e65be8 P 95a892233422466abdb0b567bcd5bb7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:53:44.147166 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.142968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.16.40.193:34099)
0812 18:53:44.143092 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.143099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.143154 (+ 55us) server_negotiation.cc:407] Connection header received
0812 18:53:44.143322 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.143325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.143381 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.143533 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:44.143790 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.144304 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.145598 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.145839 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.145932 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.146236 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.146717 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.146815 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.146923 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.146977 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 18:53:44.160458 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.156423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:57415 (local address 127.16.40.195:34009)
0812 18:53:44.156625 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.156630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.156935 (+ 305us) server_negotiation.cc:407] Connection header received
0812 18:53:44.157060 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.157064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.157124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.157209 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:44.157486 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.158274 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.159166 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.159521 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.159626 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.159960 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.160055 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.160130 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.160248 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.160312 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:53:44.160431 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.156319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34009 (local address 127.16.40.193:57415)
0812 18:53:44.156841 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:44.156854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:44.156955 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:44.157229 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:44.157231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:44.157248 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:44.157367 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.157372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.158393 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.158396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.159041 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.159050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.159653 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.159656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.159731 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.159786 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.160144 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.160151 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:53:44.160206 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 18:53:44.160279 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":127,"threads_started":1}
I20260812 18:53:44.169040 18952 raft_consensus.cc:1261] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Refusing update from remote peer 026cb669b5ca451eba23e0078cafcb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.169173 19115 raft_consensus.cc:1261] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Refusing update from remote peer 026cb669b5ca451eba23e0078cafcb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.169579 19198 consensus_queue.cc:1035] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:44.169756 19204 consensus_queue.cc:1035] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:44.178434 19225 mvcc.cc:205] Tried to move back new op lower bound from 7317753135790739456 to 7317753134811402240. Current Snapshot: MvccSnapshot[applied={T|T < 7317753135790739456}]
I20260812 18:53:44.194406 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.190454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34099 (local address 127.16.40.195:49113)
0812 18:53:44.190621 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:44.190649 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:44.190746 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:44.191002 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:44.191004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:44.191013 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:44.191128 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.191135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.192193 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.192195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.192931 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:44.192944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.193610 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.193614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:44.193702 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.193760 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.194108 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.194114 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 18:53:44.194170 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 18:53:44.194255 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 18:53:44.194887 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:44.190651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:49113 (local address 127.16.40.193:34099)
0812 18:53:44.190764 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:44.190769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:44.190801 (+ 32us) server_negotiation.cc:407] Connection header received
0812 18:53:44.190847 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:44.190850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:44.190906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:44.190989 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:44.191269 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.192071 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.193065 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:44.193411 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:44.193525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:44.193911 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:44.194002 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:44.194064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:44.194703 (+ 639us) server_negotiation.cc:299] Negotiation successful
0812 18:53:44.194765 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:53:44.228782 18605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:44.229092 18605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:44.232386 18605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Sending DeleteTablet for 3 replicas of tablet dd5697b472cf41bba352c506ef30effb
I20260812 18:53:44.233309 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5697b472cf41bba352c506ef30effb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:44 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57118
I20260812 18:53:44.233624 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5697b472cf41bba352c506ef30effb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:44 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260812 18:53:44.234346 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5697b472cf41bba352c506ef30effb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:44 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46608
I20260812 18:53:44.234987 19249 tablet_replica.cc:330] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: stopping tablet replica
I20260812 18:53:44.235093 18605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:44.235584 18605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:44.235605 19249 raft_consensus.cc:2229] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:44.236178 19249 raft_consensus.cc:2258] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:44.237104 19253 tablet_replica.cc:330] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: stopping tablet replica
I20260812 18:53:44.237424 19253 raft_consensus.cc:2229] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:44.237624 19253 raft_consensus.cc:2258] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:44.239152 19253 ts_tablet_manager.cc:1905] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:44.242295 18767 tablet_service.cc:1467] Processing CreateTablet for tablet ef09d8702b4740b496ebada025538b90 (DEFAULT_TABLE table=test-table [id=1d13841ef21e4db58ff82e477ddfec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:44.242620 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09d8702b4740b496ebada025538b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.242553 18927 tablet_service.cc:1467] Processing CreateTablet for tablet ef09d8702b4740b496ebada025538b90 (DEFAULT_TABLE table=test-table [id=1d13841ef21e4db58ff82e477ddfec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:44.242965 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09d8702b4740b496ebada025538b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.245249 19191 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Bootstrap starting.
I20260812 18:53:44.245858 19086 tablet_service.cc:1467] Processing CreateTablet for tablet ef09d8702b4740b496ebada025538b90 (DEFAULT_TABLE table=test-table [id=1d13841ef21e4db58ff82e477ddfec2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:44.246142 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef09d8702b4740b496ebada025538b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:44.246286 19191 tablet_bootstrap.cc:654] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.248754 19251 tablet_replica.cc:330] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: stopping tablet replica
I20260812 18:53:44.248860 19193 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Bootstrap starting.
I20260812 18:53:44.249157 19251 raft_consensus.cc:2229] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:44.249419 19251 raft_consensus.cc:2258] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:44.249861 19193 tablet_bootstrap.cc:654] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.250592 19249 ts_tablet_manager.cc:1905] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:44.250813 19192 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Bootstrap starting.
I20260812 18:53:44.250857 19251 ts_tablet_manager.cc:1905] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:44.251828 19192 tablet_bootstrap.cc:654] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:44.254086 19251 ts_tablet_manager.cc:1918] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:44.254204 19251 log.cc:1196] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/dd5697b472cf41bba352c506ef30effb
I20260812 18:53:44.254562 19251 ts_tablet_manager.cc:1939] T dd5697b472cf41bba352c506ef30effb P 95a892233422466abdb0b567bcd5bb7f: Deleting consensus metadata
I20260812 18:53:44.254711 19253 ts_tablet_manager.cc:1918] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:44.254807 19253 log.cc:1196] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/dd5697b472cf41bba352c506ef30effb
I20260812 18:53:44.255127 19253 ts_tablet_manager.cc:1939] T dd5697b472cf41bba352c506ef30effb P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting consensus metadata
I20260812 18:53:44.256064 18594 catalog_manager.cc:4841] TS 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957): tablet dd5697b472cf41bba352c506ef30effb (table test-table [id=4fcc0944a9ca487eb5060b150483987b]) successfully deleted
I20260812 18:53:44.256227 18597 catalog_manager.cc:4841] TS ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009): tablet dd5697b472cf41bba352c506ef30effb (table test-table [id=4fcc0944a9ca487eb5060b150483987b]) successfully deleted
I20260812 18:53:44.256565 19191 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: No bootstrap required, opened a new log
I20260812 18:53:44.256664 19191 ts_tablet_manager.cc:1397] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260812 18:53:44.257151 19191 raft_consensus.cc:348] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.257265 19191 raft_consensus.cc:374] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.257297 19191 raft_consensus.cc:729] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.257388 19192 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: No bootstrap required, opened a new log
I20260812 18:53:44.257427 19191 consensus_queue.cc:260] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.257489 19192 ts_tablet_manager.cc:1397] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 18:53:44.257700 19191 ts_tablet_manager.cc:1428] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:44.257925 19192 raft_consensus.cc:348] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.258050 19192 raft_consensus.cc:374] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.258085 19192 raft_consensus.cc:729] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a892233422466abdb0b567bcd5bb7f, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.258217 19192 consensus_queue.cc:260] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.258482 19192 ts_tablet_manager.cc:1428] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:44.260838 19249 ts_tablet_manager.cc:1918] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:44.261066 19249 log.cc:1196] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/dd5697b472cf41bba352c506ef30effb
I20260812 18:53:44.261520 19249 ts_tablet_manager.cc:1939] T dd5697b472cf41bba352c506ef30effb P 026cb669b5ca451eba23e0078cafcb5f: Deleting consensus metadata
I20260812 18:53:44.262579 19193 tablet_bootstrap.cc:492] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: No bootstrap required, opened a new log
I20260812 18:53:44.262699 19193 ts_tablet_manager.cc:1397] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260812 18:53:44.263216 19193 raft_consensus.cc:348] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.263339 19193 raft_consensus.cc:374] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:44.263370 19193 raft_consensus.cc:729] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Initialized, Role: FOLLOWER
I20260812 18:53:44.263527 19193 consensus_queue.cc:260] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.263811 19193 ts_tablet_manager.cc:1428] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 18:53:44.264433 18595 catalog_manager.cc:4841] TS 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099): tablet dd5697b472cf41bba352c506ef30effb (table test-table [id=4fcc0944a9ca487eb5060b150483987b]) successfully deleted
I20260812 18:53:44.473388 19240 raft_consensus.cc:482] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:44.473569 19240 raft_consensus.cc:504] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.474103 19240 leader_election.cc:288] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957), ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009)
I20260812 18:53:44.474676 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09d8702b4740b496ebada025538b90" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" is_pre_election: true
I20260812 18:53:44.474713 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09d8702b4740b496ebada025538b90" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f" is_pre_election: true
I20260812 18:53:44.474859 19115 raft_consensus.cc:2454] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 0.
I20260812 18:53:44.474886 18951 raft_consensus.cc:2454] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 0.
I20260812 18:53:44.475262 18713 leader_election.cc:302] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, 95a892233422466abdb0b567bcd5bb7f; no voters:
I20260812 18:53:44.475544 19240 raft_consensus.cc:2781] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:44.475628 19240 raft_consensus.cc:482] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:44.475662 19240 raft_consensus.cc:3037] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.476855 19240 raft_consensus.cc:504] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.477294 19240 leader_election.cc:288] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 election: Requested vote from peers 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957), ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009)
I20260812 18:53:44.477742 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09d8702b4740b496ebada025538b90" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0"
I20260812 18:53:44.477763 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef09d8702b4740b496ebada025538b90" candidate_uuid: "026cb669b5ca451eba23e0078cafcb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f"
I20260812 18:53:44.477912 19115 raft_consensus.cc:3037] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.477922 18951 raft_consensus.cc:3037] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:44.478937 19115 raft_consensus.cc:2454] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 1.
I20260812 18:53:44.478941 18951 raft_consensus.cc:2454] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026cb669b5ca451eba23e0078cafcb5f in term 1.
I20260812 18:53:44.479349 18713 leader_election.cc:302] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, 95a892233422466abdb0b567bcd5bb7f; no voters:
I20260812 18:53:44.479573 19240 raft_consensus.cc:2781] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:44.479666 19240 raft_consensus.cc:686] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 1 LEADER]: Becoming Leader. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Running, Role: LEADER
I20260812 18:53:44.479825 19240 consensus_queue.cc:237] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:44.481881 18610 catalog_manager.cc:5495] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193). New cstate: current_term: 1 leader_uuid: "026cb669b5ca451eba23e0078cafcb5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } health_report { overall_health: UNKNOWN } } }
I20260812 18:53:44.504592 19115 raft_consensus.cc:1261] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Refusing update from remote peer 026cb669b5ca451eba23e0078cafcb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.504709 18951 raft_consensus.cc:1261] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Refusing update from remote peer 026cb669b5ca451eba23e0078cafcb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:44.505242 19240 consensus_queue.cc:1035] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:44.505501 19204 consensus_queue.cc:1035] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 18:53:44.522622 19226 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 18:53:46.086320 18610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:46.086490 18610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:46.089326 18610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Sending DeleteTablet for 3 replicas of tablet ef09d8702b4740b496ebada025538b90
I20260812 18:53:46.090072 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet ef09d8702b4740b496ebada025538b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57118
I20260812 18:53:46.090101 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet ef09d8702b4740b496ebada025538b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260812 18:53:46.090592 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet ef09d8702b4740b496ebada025538b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46608
I20260812 18:53:46.091174 19291 tablet_replica.cc:330] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: stopping tablet replica
I20260812 18:53:46.091207 19292 tablet_replica.cc:330] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: stopping tablet replica
I20260812 18:53:46.091351 19291 raft_consensus.cc:2229] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:46.091362 19292 raft_consensus.cc:2229] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:46.091486 19292 raft_consensus.cc:2258] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.091599 19291 raft_consensus.cc:2258] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.092209 19292 ts_tablet_manager.cc:1905] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.092336 19291 ts_tablet_manager.cc:1905] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.093520 19290 tablet_replica.cc:330] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: stopping tablet replica
I20260812 18:53:46.093739 19290 raft_consensus.cc:2229] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:46.093966 19290 raft_consensus.cc:2258] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.094722 19290 ts_tablet_manager.cc:1905] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.095381 19292 ts_tablet_manager.cc:1918] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 18:53:46.095470 19292 log.cc:1196] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/ef09d8702b4740b496ebada025538b90
I20260812 18:53:46.095758 19292 ts_tablet_manager.cc:1939] T ef09d8702b4740b496ebada025538b90 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting consensus metadata
I20260812 18:53:46.096138 19291 ts_tablet_manager.cc:1918] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 18:53:46.096239 19291 log.cc:1196] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/ef09d8702b4740b496ebada025538b90
I20260812 18:53:46.096643 19291 ts_tablet_manager.cc:1939] T ef09d8702b4740b496ebada025538b90 P 026cb669b5ca451eba23e0078cafcb5f: Deleting consensus metadata
I20260812 18:53:46.096675 18597 catalog_manager.cc:4841] TS ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009): tablet ef09d8702b4740b496ebada025538b90 (table test-table [id=1d13841ef21e4db58ff82e477ddfec2e]) successfully deleted
I20260812 18:53:46.097400 18595 catalog_manager.cc:4841] TS 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099): tablet ef09d8702b4740b496ebada025538b90 (table test-table [id=1d13841ef21e4db58ff82e477ddfec2e]) successfully deleted
I20260812 18:53:46.097924 19290 ts_tablet_manager.cc:1918] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 18:53:46.098038 19290 log.cc:1196] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/ef09d8702b4740b496ebada025538b90
I20260812 18:53:46.098330 19290 ts_tablet_manager.cc:1939] T ef09d8702b4740b496ebada025538b90 P 95a892233422466abdb0b567bcd5bb7f: Deleting consensus metadata
I20260812 18:53:46.099452 18594 catalog_manager.cc:4841] TS 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957): tablet ef09d8702b4740b496ebada025538b90 (table test-table [id=1d13841ef21e4db58ff82e477ddfec2e]) successfully deleted
Aug 12 18:53:46 dist-test-slave-rnrw krb5kdc[18559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 18:53:46 dist-test-slave-rnrw krb5kdc[18559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:46.130040 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:46.114189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.16.40.254:39911)
0812 18:53:46.114541 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:46.114549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:46.114591 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:53:46.114641 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:46.114644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:46.114690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:46.114766 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:46.115696 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.116310 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.117482 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.117856 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.117972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:46.125976 (+ 8004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:46.125980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:46.125983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:46.126030 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:46.128027 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:46.128738 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:46.128742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:46.128743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:46.128842 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:46.129127 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:46.129130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:46.129131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:46.129352 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:46.129475 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:46.129763 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 18:53:46.129888 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
I20260812 18:53:46.133558 18610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:46.134123 18610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:46.139353 18767 tablet_service.cc:1467] Processing CreateTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (DEFAULT_TABLE table=test-table [id=1c344138ebb141849ecef6bec60c9d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.139438 18927 tablet_service.cc:1467] Processing CreateTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (DEFAULT_TABLE table=test-table [id=1c344138ebb141849ecef6bec60c9d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.139519 19086 tablet_service.cc:1467] Processing CreateTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (DEFAULT_TABLE table=test-table [id=1c344138ebb141849ecef6bec60c9d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.139706 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a978b8e5134f3a8ffb7fd123fe1c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.139706 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a978b8e5134f3a8ffb7fd123fe1c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.139850 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a978b8e5134f3a8ffb7fd123fe1c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.142190 19302 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Bootstrap starting.
I20260812 18:53:46.142693 19304 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Bootstrap starting.
I20260812 18:53:46.142735 19303 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Bootstrap starting.
I20260812 18:53:46.143289 19302 tablet_bootstrap.cc:654] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.143652 19303 tablet_bootstrap.cc:654] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.143652 19304 tablet_bootstrap.cc:654] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.144615 19302 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: No bootstrap required, opened a new log
I20260812 18:53:46.144716 19302 ts_tablet_manager.cc:1397] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:53:46.144912 19303 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: No bootstrap required, opened a new log
I20260812 18:53:46.144912 19304 tablet_bootstrap.cc:492] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: No bootstrap required, opened a new log
I20260812 18:53:46.145004 19304 ts_tablet_manager.cc:1397] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 18:53:46.145004 19303 ts_tablet_manager.cc:1397] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 18:53:46.145078 19302 raft_consensus.cc:348] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.145177 19302 raft_consensus.cc:374] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.145200 19302 raft_consensus.cc:729] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.145337 19302 consensus_queue.cc:260] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.145412 19304 raft_consensus.cc:348] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.145413 19303 raft_consensus.cc:348] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.145524 19304 raft_consensus.cc:374] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.145530 19303 raft_consensus.cc:374] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.145561 19304 raft_consensus.cc:729] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.145565 19303 raft_consensus.cc:729] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a892233422466abdb0b567bcd5bb7f, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.145617 19302 ts_tablet_manager.cc:1428] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 18:53:46.145718 19303 consensus_queue.cc:260] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.145718 19304 consensus_queue.cc:260] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.146095 19303 ts_tablet_manager.cc:1428] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 18:53:46.146548 19304 ts_tablet_manager.cc:1428] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 18:53:46.374298 19268 raft_consensus.cc:482] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:46.374468 19268 raft_consensus.cc:504] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.375041 19268 leader_election.cc:288] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:46.375586 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a978b8e5134f3a8ffb7fd123fe1c19" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f" is_pre_election: true
I20260812 18:53:46.375630 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a978b8e5134f3a8ffb7fd123fe1c19" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f" is_pre_election: true
I20260812 18:53:46.375764 18952 raft_consensus.cc:2454] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:46.375792 18790 raft_consensus.cc:2454] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:46.376143 19031 leader_election.cc:302] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:46.376355 19268 raft_consensus.cc:2781] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:46.376425 19268 raft_consensus.cc:482] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:46.376458 19268 raft_consensus.cc:3037] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.377558 19268 raft_consensus.cc:504] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.377875 19268 leader_election.cc:288] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Requested vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:46.378304 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a978b8e5134f3a8ffb7fd123fe1c19" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f"
I20260812 18:53:46.378337 18952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a978b8e5134f3a8ffb7fd123fe1c19" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f"
I20260812 18:53:46.378463 18790 raft_consensus.cc:3037] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.378484 18952 raft_consensus.cc:3037] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.379660 18790 raft_consensus.cc:2454] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:46.379660 18952 raft_consensus.cc:2454] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:46.379958 19031 leader_election.cc:302] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:46.380155 19268 raft_consensus.cc:2781] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:46.380255 19268 raft_consensus.cc:686] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 LEADER]: Becoming Leader. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Running, Role: LEADER
I20260812 18:53:46.380396 19268 consensus_queue.cc:237] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.382347 18610 catalog_manager.cc:5495] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } health_report { overall_health: HEALTHY } } }
I20260812 18:53:46.452766 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:46.449204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.16.40.195:34009)
0812 18:53:46.449518 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:46.449527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:46.449539 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:53:46.449590 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:46.449592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:46.449636 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:46.449731 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:46.450025 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.450576 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.451466 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.451725 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.451840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:46.452109 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:46.452193 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:46.452278 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:46.452398 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 18:53:46.452485 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":121,"threads_started":1}
I20260812 18:53:46.464516 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:46.459566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:49883 (local address 127.16.40.195:34009)
0812 18:53:46.460694 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:46.460698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:46.460710 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:53:46.460789 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:46.460792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:46.460870 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:46.460945 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:46.461468 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.462091 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.463277 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.463544 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.463685 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:46.463893 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:46.463992 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:46.464107 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:46.464300 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 18:53:46.464371 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1048}
I20260812 18:53:46.464596 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:46.459486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34009 (local address 127.16.40.195:49883)
0812 18:53:46.459796 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:46.459810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:46.459893 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:46.460987 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:46.460990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:46.461001 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:53:46.461292 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:46.461300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.462204 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.462208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:46.463147 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:46.463156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:46.463659 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:46.463663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:46.463749 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:46.463782 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:53:46.464113 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:53:46.464120 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:53:46.464198 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 18:53:46.464286 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260812 18:53:46.470785 18790 raft_consensus.cc:1261] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:46.470794 18952 raft_consensus.cc:1261] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:46.471206 19218 consensus_queue.cc:1035] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:46.471351 19268 consensus_queue.cc:1035] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:46.507320 18611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:46.507529 18611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:46.510087 18611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Sending DeleteTablet for 3 replicas of tablet f0a978b8e5134f3a8ffb7fd123fe1c19
I20260812 18:53:46.510809 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57118
I20260812 18:53:46.510897 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46608
I20260812 18:53:46.510929 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a978b8e5134f3a8ffb7fd123fe1c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:46 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260812 18:53:46.511044 19291 tablet_replica.cc:330] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: stopping tablet replica
I20260812 18:53:46.511101 19292 tablet_replica.cc:330] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: stopping tablet replica
I20260812 18:53:46.511104 19290 tablet_replica.cc:330] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: stopping tablet replica
I20260812 18:53:46.511165 19291 raft_consensus.cc:2229] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:46.511215 19292 raft_consensus.cc:2229] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:46.511215 19290 raft_consensus.cc:2229] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:46.511293 19291 raft_consensus.cc:2258] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.511336 19290 raft_consensus.cc:2258] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.511579 19292 raft_consensus.cc:2258] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:46.512303 19291 ts_tablet_manager.cc:1905] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.512432 19290 ts_tablet_manager.cc:1905] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.512847 18611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:46.513294 18611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:46.514986 19290 ts_tablet_manager.cc:1918] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:46.515048 19290 log.cc:1196] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/f0a978b8e5134f3a8ffb7fd123fe1c19
I20260812 18:53:46.515153 19291 ts_tablet_manager.cc:1918] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:46.515226 19291 log.cc:1196] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/f0a978b8e5134f3a8ffb7fd123fe1c19
I20260812 18:53:46.515256 19290 ts_tablet_manager.cc:1939] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 95a892233422466abdb0b567bcd5bb7f: Deleting consensus metadata
I20260812 18:53:46.515479 19291 ts_tablet_manager.cc:1939] T f0a978b8e5134f3a8ffb7fd123fe1c19 P 026cb669b5ca451eba23e0078cafcb5f: Deleting consensus metadata
I20260812 18:53:46.516323 18594 catalog_manager.cc:4841] TS 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957): tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (table test-table [id=1c344138ebb141849ecef6bec60c9d7b]) successfully deleted
I20260812 18:53:46.517272 19292 ts_tablet_manager.cc:1905] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:46.520512 19292 ts_tablet_manager.cc:1918] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:53:46.520512 19086 tablet_service.cc:1467] Processing CreateTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 (DEFAULT_TABLE table=test-table [id=3a007bf72a844d47b45f586138b6606a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.520691 19292 log.cc:1196] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/f0a978b8e5134f3a8ffb7fd123fe1c19
I20260812 18:53:46.520778 18767 tablet_service.cc:1467] Processing CreateTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 (DEFAULT_TABLE table=test-table [id=3a007bf72a844d47b45f586138b6606a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.520934 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc10c2c6a0b24237be05f4085551a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.520967 19292 ts_tablet_manager.cc:1939] T f0a978b8e5134f3a8ffb7fd123fe1c19 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting consensus metadata
I20260812 18:53:46.521060 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc10c2c6a0b24237be05f4085551a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.522197 18595 catalog_manager.cc:4841] TS 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099): tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (table test-table [id=1c344138ebb141849ecef6bec60c9d7b]) successfully deleted
I20260812 18:53:46.523340 19302 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Bootstrap starting.
I20260812 18:53:46.523435 19304 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f: Bootstrap starting.
I20260812 18:53:46.524196 19302 tablet_bootstrap.cc:654] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.524293 19304 tablet_bootstrap.cc:654] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.525147 18597 catalog_manager.cc:4841] TS ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009): tablet f0a978b8e5134f3a8ffb7fd123fe1c19 (table test-table [id=1c344138ebb141849ecef6bec60c9d7b]) successfully deleted
I20260812 18:53:46.525470 19302 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: No bootstrap required, opened a new log
I20260812 18:53:46.525579 19302 ts_tablet_manager.cc:1397] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 18:53:46.526038 18927 tablet_service.cc:1467] Processing CreateTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 (DEFAULT_TABLE table=test-table [id=3a007bf72a844d47b45f586138b6606a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:46.526022 19302 raft_consensus.cc:348] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.526160 19302 raft_consensus.cc:374] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.526192 19302 raft_consensus.cc:729] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.526355 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc10c2c6a0b24237be05f4085551a3d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:46.526324 19302 consensus_queue.cc:260] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.526594 19302 ts_tablet_manager.cc:1428] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:46.527266 19304 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f: No bootstrap required, opened a new log
I20260812 18:53:46.527351 19304 ts_tablet_manager.cc:1397] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260812 18:53:46.527829 19304 raft_consensus.cc:348] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.527949 19304 raft_consensus.cc:374] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.527988 19304 raft_consensus.cc:729] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026cb669b5ca451eba23e0078cafcb5f, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.528136 19304 consensus_queue.cc:260] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.528414 19304 ts_tablet_manager.cc:1428] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260812 18:53:46.529176 19303 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Bootstrap starting.
I20260812 18:53:46.530179 19303 tablet_bootstrap.cc:654] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:46.531445 19303 tablet_bootstrap.cc:492] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: No bootstrap required, opened a new log
I20260812 18:53:46.531548 19303 ts_tablet_manager.cc:1397] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 18:53:46.532080 19303 raft_consensus.cc:348] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.532212 19303 raft_consensus.cc:374] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:46.532253 19303 raft_consensus.cc:729] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a892233422466abdb0b567bcd5bb7f, State: Initialized, Role: FOLLOWER
I20260812 18:53:46.532389 19303 consensus_queue.cc:260] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.532734 19303 ts_tablet_manager.cc:1428] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:53:46.565754 19311 raft_consensus.cc:482] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:46.565936 19311 raft_consensus.cc:504] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.566545 19311 leader_election.cc:288] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:46.567054 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc10c2c6a0b24237be05f4085551a3d2" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f" is_pre_election: true
I20260812 18:53:46.567085 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc10c2c6a0b24237be05f4085551a3d2" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f" is_pre_election: true
I20260812 18:53:46.567235 18790 raft_consensus.cc:2454] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:46.567271 18951 raft_consensus.cc:2454] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 0.
I20260812 18:53:46.567675 19031 leader_election.cc:302] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 026cb669b5ca451eba23e0078cafcb5f, 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:46.567934 19311 raft_consensus.cc:2781] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:46.567983 19311 raft_consensus.cc:482] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:46.568011 19311 raft_consensus.cc:3037] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.569111 19311 raft_consensus.cc:504] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.569527 19311 leader_election.cc:288] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Requested vote from peers 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099), 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957)
I20260812 18:53:46.569914 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc10c2c6a0b24237be05f4085551a3d2" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026cb669b5ca451eba23e0078cafcb5f"
I20260812 18:53:46.569939 18951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc10c2c6a0b24237be05f4085551a3d2" candidate_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95a892233422466abdb0b567bcd5bb7f"
I20260812 18:53:46.570072 18790 raft_consensus.cc:3037] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.570083 18951 raft_consensus.cc:3037] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:46.571280 18790 raft_consensus.cc:2454] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:46.571309 18951 raft_consensus.cc:2454] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc4ab0bc05c4a52991a3cf8aba3f2a0 in term 1.
I20260812 18:53:46.571658 19031 leader_election.cc:302] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95a892233422466abdb0b567bcd5bb7f, ddc4ab0bc05c4a52991a3cf8aba3f2a0; no voters:
I20260812 18:53:46.571885 19311 raft_consensus.cc:2781] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:46.572257 19311 raft_consensus.cc:686] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 LEADER]: Becoming Leader. State: Replica: ddc4ab0bc05c4a52991a3cf8aba3f2a0, State: Running, Role: LEADER
I20260812 18:53:46.572407 19311 consensus_queue.cc:237] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } }
I20260812 18:53:46.574651 18610 catalog_manager.cc:5495] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc4ab0bc05c4a52991a3cf8aba3f2a0" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34009 } health_report { overall_health: HEALTHY } } }
I20260812 18:53:46.597443 18790 raft_consensus.cc:1261] T dc10c2c6a0b24237be05f4085551a3d2 P 026cb669b5ca451eba23e0078cafcb5f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:46.597661 18952 raft_consensus.cc:1261] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Refusing update from remote peer ddc4ab0bc05c4a52991a3cf8aba3f2a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:46.597952 19311 consensus_queue.cc:1035] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026cb669b5ca451eba23e0078cafcb5f" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:46.598256 19218 consensus_queue.cc:1035] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95a892233422466abdb0b567bcd5bb7f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 18:53:46.616104 19266 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 18:53:48.172942 18610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:48.173111 18610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:48.175928 18610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e9af9e31579499a8a50cf95d2b08c8d: Sending DeleteTablet for 3 replicas of tablet dc10c2c6a0b24237be05f4085551a3d2
I20260812 18:53:48.176759 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:48 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:54494
I20260812 18:53:48.176761 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:48 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57118
I20260812 18:53:48.176791 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet dc10c2c6a0b24237be05f4085551a3d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:48 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46608
I20260812 18:53:48.177222 19349 tablet_replica.cc:330] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: stopping tablet replica
I20260812 18:53:48.177282 19348 tablet_replica.cc:330] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: stopping tablet replica
I20260812 18:53:48.177407 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 18694
I20260812 18:53:48.177461 19348 raft_consensus.cc:2229] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:48.177448 19349 raft_consensus.cc:2229] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:48.179991 19349 raft_consensus.cc:2258] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:48.180840 19348 raft_consensus.cc:2258] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:48.180861 19349 ts_tablet_manager.cc:1905] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:48.181752 19348 ts_tablet_manager.cc:1905] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:48.184114 19349 ts_tablet_manager.cc:1918] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 18:53:48.184199 19349 log.cc:1196] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/dc10c2c6a0b24237be05f4085551a3d2
I20260812 18:53:48.184597 19349 ts_tablet_manager.cc:1939] T dc10c2c6a0b24237be05f4085551a3d2 P ddc4ab0bc05c4a52991a3cf8aba3f2a0: Deleting consensus metadata
I20260812 18:53:48.184791 19348 ts_tablet_manager.cc:1918] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 18:53:48.184864 19348 log.cc:1196] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/dc10c2c6a0b24237be05f4085551a3d2
I20260812 18:53:48.185158 19348 ts_tablet_manager.cc:1939] T dc10c2c6a0b24237be05f4085551a3d2 P 95a892233422466abdb0b567bcd5bb7f: Deleting consensus metadata
I20260812 18:53:48.186343 18594 catalog_manager.cc:4841] TS 95a892233422466abdb0b567bcd5bb7f (127.16.40.194:42957): tablet dc10c2c6a0b24237be05f4085551a3d2 (table test-table [id=3a007bf72a844d47b45f586138b6606a]) successfully deleted
I20260812 18:53:48.187217 18597 catalog_manager.cc:4841] TS ddc4ab0bc05c4a52991a3cf8aba3f2a0 (127.16.40.195:34009): tablet dc10c2c6a0b24237be05f4085551a3d2 (table test-table [id=3a007bf72a844d47b45f586138b6606a]) successfully deleted
W20260812 18:53:48.197606 18595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:34099 (error 108)
I20260812 18:53:48.197846 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 18854
I20260812 18:53:48.207050 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:48.206398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34099 (local address 127.0.0.1:57126)
0812 18:53:48.206765 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:48.206885 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
W20260812 18:53:48.207263 18595 catalog_manager.cc:4584] TS 026cb669b5ca451eba23e0078cafcb5f (127.16.40.193:34099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc10c2c6a0b24237be05f4085551a3d2: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34099: connect: Connection refused (error 111)
I20260812 18:53:48.211647 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19013
I20260812 18:53:48.222074 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 18573
2026-08-12T18:53:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6820 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:48 dist-test-slave-rnrw krb5kdc[19355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:48 dist-test-slave-rnrw krb5kdc[19355](info): set up 2 sockets
Aug 12 18:53:48 dist-test-slave-rnrw krb5kdc[19355](info): commencing operation
krb5kdc: starting...
W20260812 18:53:49.357529 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:49 dist-test-slave-rnrw krb5kdc[19355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:49.501169 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36911
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:49.621336 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:49.621634 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:49.621680 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:49.625869 19370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:49.625941 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:49.625962 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:49.625980 19370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:49.625995 19370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:49.631492 19370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:49.632653 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:49.633687 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:49.643118 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:49.643165 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:49.643175 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:49.643678 19370 server_base.cc:1034] running on GCE node
I20260812 18:53:49.644322 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:49.645582 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:49.646796 19370 hybrid_clock.cc:648] HybridClock initialized: now 1786560829646763 us; error 45 us; skew 500 ppm
Aug 12 18:53:49 dist-test-slave-rnrw krb5kdc[19355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560829, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:49.649941 19370 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:49.650985 19370 webserver.cc:460] Webserver started at http://127.16.40.254:42553/ using document root <none> and password file <none>
I20260812 18:53:49.651608 19370 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:49.651705 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:49.651964 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:49.654481 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "4b2b89a378dc4da1a3fbf6b19847c32d"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "0e5e48f4956d336d32f92acf13414b34"
server_key_iv: "5861e1eab214de79c3f79d8a01fb9464"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.655102 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "4b2b89a378dc4da1a3fbf6b19847c32d"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "0e5e48f4956d336d32f92acf13414b34"
server_key_iv: "5861e1eab214de79c3f79d8a01fb9464"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.659197 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:49.661924 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:49.663218 19370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 18:53:49.663374 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "4b2b89a378dc4da1a3fbf6b19847c32d"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "0e5e48f4956d336d32f92acf13414b34"
server_key_iv: "5861e1eab214de79c3f79d8a01fb9464"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.663498 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:49.682993 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:49.686239 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:49.686473 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:49.694196 19370 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:36911
I20260812 18:53:49.694226 19449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:36911 every 8 connection(s)
I20260812 18:53:49.695264 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:49.698227 19450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:49.699800 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19370
I20260812 18:53:49.699894 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:49.700181 16547 external_mini_cluster.cc:1428] Setting key 247462debf47194718d300e5396b611e
I20260812 18:53:49.704420 19450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d: Bootstrap starting.
I20260812 18:53:49.706724 19450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:49.707553 19450 log.cc:824] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:53:49 dist-test-slave-rnrw krb5kdc[19355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:49.709817 19450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d: No bootstrap required, opened a new log
I20260812 18:53:49.714017 19450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } }
I20260812 18:53:49.714310 19450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:49.714450 19450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2b89a378dc4da1a3fbf6b19847c32d, State: Initialized, Role: FOLLOWER
I20260812 18:53:49.714948 19450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } }
I20260812 18:53:49.715107 19450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:49.715197 19450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:49.715324 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:49.701743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.16.40.254:36911)
0812 18:53:49.702206 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:49.702221 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:49.702258 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:53:49.702877 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:49.702893 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:49.703228 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:49.703568 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:49.704442 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:49.705502 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:49.706389 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:49.706932 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:49.707124 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:49.709960 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:49.709969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:49.709995 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:49.710024 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:49.711774 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:49.712298 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:49.712303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:49.712309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:49.712421 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:49.712754 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:49.712758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:49.712760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:49.713388 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:49.713729 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:49.714050 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 18:53:49.714418 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":131,"threads_started":1}
I20260812 18:53:49.715327 19450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:49.716770 19450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } }
I20260812 18:53:49.717123 19450 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2b89a378dc4da1a3fbf6b19847c32d; no voters:
I20260812 18:53:49.717453 19450 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:49.717558 19456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:49.717804 19456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [term 1 LEADER]: Becoming Leader. State: Replica: 4b2b89a378dc4da1a3fbf6b19847c32d, State: Running, Role: LEADER
I20260812 18:53:49.718179 19456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } }
I20260812 18:53:49.718780 19450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:49.719980 19458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2b89a378dc4da1a3fbf6b19847c32d. Latest consensus state: current_term: 1 leader_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } } }
I20260812 18:53:49.720119 19458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:49.720405 19457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b89a378dc4da1a3fbf6b19847c32d" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36911 } } }
I20260812 18:53:49.720515 19466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:49.720633 19457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:49.723908 19466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:49.730268 19466 catalog_manager.cc:1347] Generated new cluster ID: 3706c70c2f0742f99f5a31a32184fbbd
I20260812 18:53:49.730360 19466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:49.776990 19466 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:49.778486 19466 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:49.789511 19466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2b89a378dc4da1a3fbf6b19847c32d: Generated new TSK 0
I20260812 18:53:49.790489 19466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:49.836826 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:49.954278 19479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:49.954638 19479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:49.954746 19479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:49.959245 19479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:49.959370 19479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:49.959472 19479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:49.965286 19479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:49.966512 19479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:49.967686 19479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:49.978190 19488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:49.978207 19485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:49.978207 19486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:49.978847 19479 server_base.cc:1034] running on GCE node
I20260812 18:53:49.979333 19479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:49.980005 19479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:49.981194 19479 hybrid_clock.cc:648] HybridClock initialized: now 1786560829981176 us; error 31 us; skew 500 ppm
Aug 12 18:53:49 dist-test-slave-rnrw krb5kdc[19355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560829, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:49.984131 19479 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:49.985184 19479 webserver.cc:460] Webserver started at http://127.16.40.193:39355/ using document root <none> and password file <none>
I20260812 18:53:49.985785 19479 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:49.985877 19479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:49.986135 19479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:49.988268 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "e1290c574fc84d60aae35d49931e913b"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "2d04889dcbf2415bfe565a9ba71255e0"
server_key_iv: "7dd489b31af39cb7bc584eb679821754"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.988941 19479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "e1290c574fc84d60aae35d49931e913b"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "2d04889dcbf2415bfe565a9ba71255e0"
server_key_iv: "7dd489b31af39cb7bc584eb679821754"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.992996 19479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:53:49.995694 19497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:49.997118 19479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:49.997246 19479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "e1290c574fc84d60aae35d49931e913b"
format_stamp: "Formatted at 2026-08-12 18:53:49 on dist-test-slave-rnrw"
server_key: "2d04889dcbf2415bfe565a9ba71255e0"
server_key_iv: "7dd489b31af39cb7bc584eb679821754"
server_key_version: "encryptionkey@0"
I20260812 18:53:49.997363 19479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:50.025228 19479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:50.028910 19479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:50.029134 19479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:50.029790 19479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:50.030682 19479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:50.030731 19479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.030803 19479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:50.030836 19479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.043785 19479 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:33929
I20260812 18:53:50.043804 19629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:33929 every 8 connection(s)
I20260812 18:53:50.045463 19479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:50.047037 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19479
I20260812 18:53:50.047166 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:50.047523 16547 external_mini_cluster.cc:1428] Setting key 072ea2b7e1d86b71d47c70b18d387fca
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560829, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:53:50.069924 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.047972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:36723 (local address 127.16.40.254:36911)
0812 18:53:50.048142 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:50.048146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:50.049396 (+ 1250us) server_negotiation.cc:407] Connection header received
0812 18:53:50.051327 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:50.051331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:50.051391 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:50.051494 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:50.053213 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.054062 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.055918 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.056156 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.056269 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.059736 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:50.059739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:50.059741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:50.059797 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:50.066530 (+ 6733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.067558 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.067565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.067568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.067691 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.068211 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.068215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.068217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.068532 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:50.068661 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:50.069596 (+ 935us) server_negotiation.cc:299] Negotiation successful
0812 18:53:50.069738 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:53:50.071237 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.048635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36911 (local address 127.16.40.193:36723)
0812 18:53:50.049233 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:50.049280 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:50.051036 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:50.051688 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:50.051700 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:50.052277 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:50.053030 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.053055 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.054222 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.054229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.055782 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.055792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.056329 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.056333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.056508 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.057248 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:50.057276 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:50.059479 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:50.066756 (+ 7277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.066766 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.066791 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.067406 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.067808 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.067815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.067820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.068060 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.068746 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:50.068756 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:50.069259 (+ 503us) client_negotiation.cc:769] Sending connection context
0812 18:53:50.069525 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 18:53:50.069906 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":167,"threads_started":1}
I20260812 18:53:50.072839 19630 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36911
I20260812 18:53:50.073151 19630 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:50.073799 19630 heartbeater.cc:511] Master 127.16.40.254:36911 requested a full tablet report, sending...
I20260812 18:53:50.075423 19410 ts_manager.cc:194] Registered new tserver with Master: e1290c574fc84d60aae35d49931e913b (127.16.40.193:33929)
I20260812 18:53:50.076812 19410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:36723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:50.088075 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:50.208472 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:50.208767 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:50.208856 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:50.212962 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:50.213030 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:50.213120 19636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:50.218562 19636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:50.219702 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:50.220887 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:50.230803 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:50.230814 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:50.230854 19636 server_base.cc:1034] running on GCE node
W20260812 18:53:50.230816 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:50.231436 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:50.232136 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:50.233309 19636 hybrid_clock.cc:648] HybridClock initialized: now 1786560830233291 us; error 30 us; skew 500 ppm
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:50.236217 19636 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:50.237236 19636 webserver.cc:460] Webserver started at http://127.16.40.194:38625/ using document root <none> and password file <none>
I20260812 18:53:50.237893 19636 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:50.237969 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:50.238204 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:50.240482 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "7d3428f7ad7f4ef49f5aba9482c346e2"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "a0edf6045b73d2ca631a354f0138c572"
server_key_iv: "475eee5125eafef741002119193a46ed"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.241169 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "7d3428f7ad7f4ef49f5aba9482c346e2"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "a0edf6045b73d2ca631a354f0138c572"
server_key_iv: "475eee5125eafef741002119193a46ed"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.245124 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:53:50.247769 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.249246 19636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:50.249380 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "7d3428f7ad7f4ef49f5aba9482c346e2"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "a0edf6045b73d2ca631a354f0138c572"
server_key_iv: "475eee5125eafef741002119193a46ed"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.249511 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:50.264242 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:50.267813 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:50.268088 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:50.268824 19636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:50.269739 19636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:50.269790 19636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.269855 19636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:50.269888 19636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.279520 19636 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:38711
I20260812 18:53:50.279531 19784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:38711 every 8 connection(s)
I20260812 18:53:50.280483 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:50.287626 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19636
I20260812 18:53:50.287760 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:50.288022 16547 external_mini_cluster.cc:1428] Setting key 8ac7dc2e7159f8e049301f652b12ef58
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:50.296160 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.282484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:39507 (local address 127.16.40.254:36911)
0812 18:53:50.282656 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:50.282660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:50.283486 (+ 826us) server_negotiation.cc:407] Connection header received
0812 18:53:50.284843 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:50.284846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:50.284910 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:50.285009 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:50.286786 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.287342 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.288751 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.288991 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.289117 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.292237 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:50.292242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:50.292244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:50.292270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:50.293816 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.294433 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.294437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.294440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.294530 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.294944 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.294948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.294949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.295156 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:50.295271 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:50.295829 (+ 558us) server_negotiation.cc:299] Negotiation successful
0812 18:53:50.295984 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:53:50.297248 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.282866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36911 (local address 127.16.40.194:39507)
0812 18:53:50.283330 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:50.283374 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:50.284623 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:50.285213 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:50.285220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:50.285783 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:50.286631 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.286647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.287510 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.287513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.288609 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.288622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.289180 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.289185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.289374 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.290126 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:50.290143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:50.292040 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:50.294000 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.294007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.294025 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.294310 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.294629 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.294633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.294635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.294837 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.295276 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:50.295285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:50.295579 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 18:53:50.295787 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 18:53:50.296160 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260812 18:53:50.298559 19785 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36911
I20260812 18:53:50.298834 19785 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:50.299454 19785 heartbeater.cc:511] Master 127.16.40.254:36911 requested a full tablet report, sending...
I20260812 18:53:50.300675 19410 ts_manager.cc:194] Registered new tserver with Master: 7d3428f7ad7f4ef49f5aba9482c346e2 (127.16.40.194:38711)
I20260812 18:53:50.301271 19410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:39507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:50.319557 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:50.439831 19790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:50.440124 19790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:50.440197 19790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:50.444638 19790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:50.444715 19790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:50.444808 19790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:50.450646 19790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:50.451895 19790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:50.453078 19790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:50.463698 19799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:50.463729 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:50.463698 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:50.464107 19790 server_base.cc:1034] running on GCE node
I20260812 18:53:50.464674 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:50.465469 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:50.466670 19790 hybrid_clock.cc:648] HybridClock initialized: now 1786560830466651 us; error 33 us; skew 500 ppm
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:50.469739 19790 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:50.470690 19790 webserver.cc:460] Webserver started at http://127.16.40.195:34561/ using document root <none> and password file <none>
I20260812 18:53:50.471297 19790 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:50.471393 19790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:50.471640 19790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:50.473860 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "741f5e6934f4444fab1f35c2c112be57"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "5f2fc895fa053c4a5e96235f45aaeea6"
server_key_iv: "fd97b918f2e1cabde47813629ca60eaf"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.474463 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "741f5e6934f4444fab1f35c2c112be57"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "5f2fc895fa053c4a5e96235f45aaeea6"
server_key_iv: "fd97b918f2e1cabde47813629ca60eaf"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.478407 19790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 18:53:50.481067 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.482352 19790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:50.482501 19790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "741f5e6934f4444fab1f35c2c112be57"
format_stamp: "Formatted at 2026-08-12 18:53:50 on dist-test-slave-rnrw"
server_key: "5f2fc895fa053c4a5e96235f45aaeea6"
server_key_iv: "fd97b918f2e1cabde47813629ca60eaf"
server_key_version: "encryptionkey@0"
I20260812 18:53:50.482626 19790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:50.492959 19790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:50.496479 19790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:50.496742 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:50.497465 19790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:50.498443 19790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:50.498517 19790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.498593 19790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:50.498644 19790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:50.508476 19790 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:33755
I20260812 18:53:50.508500 19933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:33755 every 8 connection(s)
I20260812 18:53:50.509613 19790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:50.517900 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19790
I20260812 18:53:50.518050 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:50.518383 16547 external_mini_cluster.cc:1428] Setting key 7505e2bfd02f166074bc09756f80c48c
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560830, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:50.526296 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.511791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:57157 (local address 127.16.40.254:36911)
0812 18:53:50.511974 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:50.511978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:50.513033 (+ 1055us) server_negotiation.cc:407] Connection header received
0812 18:53:50.514433 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:50.514437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:50.514502 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:50.514574 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:50.515950 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.516518 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.517941 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.518274 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.518377 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.522040 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:50.522043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:50.522045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:50.522090 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:50.523667 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.524302 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.524305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.524306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.524391 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:50.524850 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:50.524852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:50.524854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:50.525055 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:50.525146 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:50.526014 (+ 868us) server_negotiation.cc:299] Negotiation successful
0812 18:53:50.526156 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:53:50.527231 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:50.512201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36911 (local address 127.16.40.195:57157)
0812 18:53:50.512846 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:50.512896 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:50.514190 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:50.514796 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:50.514803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:50.515209 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:50.515748 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.515762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.516720 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.516723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.517780 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:50.517794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:50.518479 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:50.518484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:50.518790 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:50.519577 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:50.519602 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:50.521834 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:50.523838 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.523846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.523864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.524194 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.524485 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:50.524488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:50.524489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:50.524757 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:50.525192 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:50.525199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:50.525490 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 18:53:50.525874 (+ 384us) client_negotiation.cc:240] Negotiation successful
0812 18:53:50.526228 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":134,"threads_started":1}
I20260812 18:53:50.528985 19935 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36911
I20260812 18:53:50.529321 19935 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:50.529940 19935 heartbeater.cc:511] Master 127.16.40.254:36911 requested a full tablet report, sending...
I20260812 18:53:50.531273 19410 ts_manager.cc:194] Registered new tserver with Master: 741f5e6934f4444fab1f35c2c112be57 (127.16.40.195:33755)
I20260812 18:53:50.532136 19410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:57157
I20260812 18:53:50.533478 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 18:53:50.550894 19454 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:53:50.549241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.16.40.254:36911)
0812 18:53:50.549432 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:50.549438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:50.549627 (+ 189us) server_negotiation.cc:407] Connection header received
0812 18:53:50.549773 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:50.549776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:50.549822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:50.549914 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:50.550771 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57600: BlockingRecv error: recv got EOF from 127.0.0.1:57600 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 18:53:50.552126 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19479
I20260812 18:53:50.560856 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19636
I20260812 18:53:50.570530 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19790
I20260812 18:53:50.581290 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19370
2026-08-12T18:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2357 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19950](info): set up 2 sockets
Aug 12 18:53:50 dist-test-slave-rnrw krb5kdc[19950](info): commencing operation
krb5kdc: starting...
W20260812 18:53:51.733354 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:51 dist-test-slave-rnrw krb5kdc[19950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:51.876116 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:46651
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:46651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:51.995414 19967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:51.995769 19967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:51.995862 19967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:52.000113 19967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:52.000228 19967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:52.000267 19967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:52.000320 19967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:52.000376 19967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:52.006099 19967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:46651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:46651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:52.007352 19967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:52.008499 19967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:52.017408 19973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:52.017422 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:52.017400 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:52.017674 19967 server_base.cc:1034] running on GCE node
I20260812 18:53:52.018494 19967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:52.019714 19967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:52.020908 19967 hybrid_clock.cc:648] HybridClock initialized: now 1786560832020864 us; error 50 us; skew 500 ppm
Aug 12 18:53:52 dist-test-slave-rnrw krb5kdc[19950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560832, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:52.032307 19967 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:52.033524 19967 webserver.cc:460] Webserver started at http://127.16.40.254:39363/ using document root <none> and password file <none>
I20260812 18:53:52.034176 19967 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:52.034247 19967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:52.034438 19967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:52.036892 19967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a"
format_stamp: "Formatted at 2026-08-12 18:53:52 on dist-test-slave-rnrw"
server_key: "0958e297194cc04eda2663494a7b6c1e"
server_key_iv: "b1bb6507b95e1e67620817359bf6edfd"
server_key_version: "encryptionkey@0"
I20260812 18:53:52.037510 19967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a"
format_stamp: "Formatted at 2026-08-12 18:53:52 on dist-test-slave-rnrw"
server_key: "0958e297194cc04eda2663494a7b6c1e"
server_key_iv: "b1bb6507b95e1e67620817359bf6edfd"
server_key_version: "encryptionkey@0"
I20260812 18:53:52.041954 19967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:53:52.045187 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:52.046525 19967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:53:52.046737 19967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a"
format_stamp: "Formatted at 2026-08-12 18:53:52 on dist-test-slave-rnrw"
server_key: "0958e297194cc04eda2663494a7b6c1e"
server_key_iv: "b1bb6507b95e1e67620817359bf6edfd"
server_key_version: "encryptionkey@0"
I20260812 18:53:52.046886 19967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:52.062674 19967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:52.066143 19967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:52.066377 19967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:52.074602 19967 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:46651
I20260812 18:53:52.074996 20053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:46651 every 8 connection(s)
I20260812 18:53:52.075830 19967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:52.078985 20054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:52.084381 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 19967
I20260812 18:53:52.084497 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:52.084838 16547 external_mini_cluster.cc:1428] Setting key 2372c8bd3366ea64f00c496360514634
I20260812 18:53:52.085101 20054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a: Bootstrap starting.
I20260812 18:53:52.087869 20054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:52.088733 20054 log.cc:824] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:52.091082 20054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a: No bootstrap required, opened a new log
Aug 12 18:53:52 dist-test-slave-rnrw krb5kdc[19950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:52.095547 20054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } }
I20260812 18:53:52.095873 20054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:52.095948 20054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae772bf8bbb4bf69f0f21c45cdd5b0a, State: Initialized, Role: FOLLOWER
I20260812 18:53:52.096527 20054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } }
I20260812 18:53:52.096735 20054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:52.096807 20054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:52.096923 20054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:52.098179 20054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } }
I20260812 18:53:52.098583 20054 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae772bf8bbb4bf69f0f21c45cdd5b0a; no voters:
I20260812 18:53:52.099035 20054 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:52.099176 20060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:52.099422 20060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [term 1 LEADER]: Becoming Leader. State: Replica: 4ae772bf8bbb4bf69f0f21c45cdd5b0a, State: Running, Role: LEADER
I20260812 18:53:52.099753 20060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } }
I20260812 18:53:52.100517 20054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:52.101536 20063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae772bf8bbb4bf69f0f21c45cdd5b0a. Latest consensus state: current_term: 1 leader_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } } }
I20260812 18:53:52.101655 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:52.086254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.16.40.254:46651)
0812 18:53:52.086770 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:52.086786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:52.086822 (+ 36us) server_negotiation.cc:407] Connection header received
0812 18:53:52.087499 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:52.087517 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:52.087859 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:52.088237 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:52.089229 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:52.090657 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:52.091847 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:52.092566 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:52.092732 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:52.095270 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:52.095276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:52.095295 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:52.095329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:52.097532 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:52.098031 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:52.098041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:52.098050 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:52.098209 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:52.098574 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:52.098578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:52.098579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:52.099191 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:52.099410 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:52.100203 (+ 793us) server_negotiation.cc:299] Negotiation successful
0812 18:53:52.100535 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":164,"threads_started":1}
I20260812 18:53:52.101819 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae772bf8bbb4bf69f0f21c45cdd5b0a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 46651 } } }
I20260812 18:53:52.101893 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:52.101675 20063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:52.102519 20075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:52.105201 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:53:52.105981 20075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:52.112515 20075 catalog_manager.cc:1347] Generated new cluster ID: ab6a5ad00c8f465299a576fb43a167e9
I20260812 18:53:52.112641 20075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:52.132946 20075 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:52.134227 20075 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:52.135048 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:52.128890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.16.40.254:46651)
0812 18:53:52.129046 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:52.129051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:52.129508 (+ 457us) server_negotiation.cc:407] Connection header received
0812 18:53:52.129685 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:52.129688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:52.129750 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:52.129840 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:52.130442 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:52.131037 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:52.132331 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:52.132678 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:52.132799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:52.133846 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:52.133850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:52.133853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:53:52.134499 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:52.134539 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:53:52.134560 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:52.134661 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:52.134804 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 18:53:52.134894 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":53}
I20260812 18:53:52.140777 20075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae772bf8bbb4bf69f0f21c45cdd5b0a: Generated new TSK 0
I20260812 18:53:52.141677 20075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:53:52.153110 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 19967
2026-08-12T18:53:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1572 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:52 dist-test-slave-rnrw krb5kdc[20098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:52 dist-test-slave-rnrw krb5kdc[20098](info): set up 2 sockets
Aug 12 18:53:52 dist-test-slave-rnrw krb5kdc[20098](info): commencing operation
krb5kdc: starting...
W20260812 18:53:53.355787 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:53 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:53.501363 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35535
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:53.624871 20115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:53.625204 20115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:53.625250 20115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:53.629871 20115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:53.629956 20115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:53.629977 20115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:53.629999 20115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:53.630015 20115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:53.636093 20115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:53.637424 20115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:53.638731 20115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:53.648310 20123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:53.648311 20122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:53.648401 20126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:53.649047 20115 server_base.cc:1034] running on GCE node
I20260812 18:53:53.649758 20115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:53.651125 20115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:53.652359 20115 hybrid_clock.cc:648] HybridClock initialized: now 1786560833652351 us; error 46 us; skew 500 ppm
Aug 12 18:53:53 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:53.655795 20115 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:53.656966 20115 webserver.cc:460] Webserver started at http://127.16.40.254:33663/ using document root <none> and password file <none>
I20260812 18:53:53.657672 20115 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:53.657784 20115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:53.658058 20115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:53.660384 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "00a7b0a77aee4d0581ad7ebbcb160c77"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "e9250c77c263efbe9697a6da5fae6c0c"
server_key_iv: "6496a838631239ad17dc131d5864a5bb"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.661096 20115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "00a7b0a77aee4d0581ad7ebbcb160c77"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "e9250c77c263efbe9697a6da5fae6c0c"
server_key_iv: "6496a838631239ad17dc131d5864a5bb"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.665568 20115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:53.668392 20134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:53.670037 20115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:53.670229 20115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "00a7b0a77aee4d0581ad7ebbcb160c77"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "e9250c77c263efbe9697a6da5fae6c0c"
server_key_iv: "6496a838631239ad17dc131d5864a5bb"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.670365 20115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:53.684965 20115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:53.688767 20115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:53.689007 20115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:53.696965 20115 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:35535
I20260812 18:53:53.697306 20194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:35535 every 8 connection(s)
I20260812 18:53:53.698079 20115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:53.699925 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20115
I20260812 18:53:53.700066 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:53.700346 16547 external_mini_cluster.cc:1428] Setting key c30f265de849c594bcbd8cf075844626
I20260812 18:53:53.701393 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:53.707727 20195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77: Bootstrap starting.
Aug 12 18:53:53 dist-test-slave-rnrw krb5kdc[20098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:53.710522 20195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:53.711573 20195 log.cc:824] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:53.713896 20195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77: No bootstrap required, opened a new log
I20260812 18:53:53.716264 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:53.701997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.16.40.254:35535)
0812 18:53:53.702508 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:53.702524 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:53.702563 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:53:53.703243 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:53.703260 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:53.703623 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:53.704022 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:53.704917 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.705935 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.706836 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.707345 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.707533 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:53.710369 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:53.710375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:53.710392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:53.710425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:53.712677 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:53.713246 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:53.713262 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:53.713268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:53.713401 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:53.713766 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:53.713770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:53.713774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:53.714302 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:53.714536 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:53.715087 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 18:53:53.715420 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":169,"threads_started":1}
I20260812 18:53:53.717941 20195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } }
I20260812 18:53:53.718235 20195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:53.718325 20195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a7b0a77aee4d0581ad7ebbcb160c77, State: Initialized, Role: FOLLOWER
I20260812 18:53:53.718842 20195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } }
I20260812 18:53:53.719036 20195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:53.719120 20195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:53.719229 20195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:53.720273 20195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } }
I20260812 18:53:53.720713 20195 leader_election.cc:302] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a7b0a77aee4d0581ad7ebbcb160c77; no voters:
I20260812 18:53:53.721097 20195 leader_election.cc:288] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:53.721231 20202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:53.721539 20202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [term 1 LEADER]: Becoming Leader. State: Replica: 00a7b0a77aee4d0581ad7ebbcb160c77, State: Running, Role: LEADER
I20260812 18:53:53.721881 20202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } }
I20260812 18:53:53.722512 20195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:53.723490 20204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a7b0a77aee4d0581ad7ebbcb160c77. Latest consensus state: current_term: 1 leader_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } } }
I20260812 18:53:53.723626 20204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:53.724025 20203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a7b0a77aee4d0581ad7ebbcb160c77" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35535 } } }
I20260812 18:53:53.724179 20203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:53.725069 20213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:53.728349 20213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:53.734766 20213 catalog_manager.cc:1347] Generated new cluster ID: d7643734072949ba8216d7bbead75ee1
I20260812 18:53:53.734862 20213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:53.745527 20213 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:53.746882 20213 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:53.753746 20213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a7b0a77aee4d0581ad7ebbcb160c77: Generated new TSK 0
I20260812 18:53:53.754676 20213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:53.795985 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:53.915735 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:53.916098 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:53.916180 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:53.920382 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:53.920516 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:53.920643 20229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:53.926568 20229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:53.927819 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:53.928988 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:53.939592 20243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:53.939716 20229 server_base.cc:1034] running on GCE node
W20260812 18:53:53.939628 20238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:53.939555 20239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:53.940286 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:53.941210 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:53.942399 20229 hybrid_clock.cc:648] HybridClock initialized: now 1786560833942376 us; error 37 us; skew 500 ppm
Aug 12 18:53:53 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:53.945464 20229 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:53.946427 20229 webserver.cc:460] Webserver started at http://127.16.40.193:40429/ using document root <none> and password file <none>
I20260812 18:53:53.947038 20229 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:53.947111 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:53.947340 20229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:53.949643 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "d352159eda284cc38deb03beb1fce1dd"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "6f49f45559a62cd2d6e1d228f9f766c0"
server_key_iv: "a28ec5a4dd2924d19b5a71c22e21815b"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.950218 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "d352159eda284cc38deb03beb1fce1dd"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "6f49f45559a62cd2d6e1d228f9f766c0"
server_key_iv: "a28ec5a4dd2924d19b5a71c22e21815b"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.954205 20229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:53.956981 20250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:53.958587 20229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:53:53.958847 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "d352159eda284cc38deb03beb1fce1dd"
format_stamp: "Formatted at 2026-08-12 18:53:53 on dist-test-slave-rnrw"
server_key: "6f49f45559a62cd2d6e1d228f9f766c0"
server_key_iv: "a28ec5a4dd2924d19b5a71c22e21815b"
server_key_version: "encryptionkey@0"
I20260812 18:53:53.958983 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:53.970470 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:53.974255 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:53.974479 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:53.975167 20229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:53.976079 20229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:53.976161 20229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:53.976233 20229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:53.976287 20229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:53.985625 20229 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:36069
I20260812 18:53:53.985679 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:36069 every 8 connection(s)
I20260812 18:53:53.986652 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:53.994614 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20229
I20260812 18:53:53.994760 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:53.995064 16547 external_mini_cluster.cc:1428] Setting key 4563de7f738c06f8fccbf802d3dd4cea
Aug 12 18:53:53 dist-test-slave-rnrw krb5kdc[20098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560833, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:54.002642 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:53.988669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:35143 (local address 127.16.40.254:35535)
0812 18:53:53.988824 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:53.988828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:53.989670 (+ 842us) server_negotiation.cc:407] Connection header received
0812 18:53:53.991179 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:53.991182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:53.991248 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:53.991370 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:53.992698 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.993305 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.994566 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.994923 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.995048 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:53.998092 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:53.998096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:53.998099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:53.998136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:54.000012 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.000661 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.000667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.000672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.000773 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.001204 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.001211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.001215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.001501 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:54.001615 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:54.002219 (+ 604us) server_negotiation.cc:299] Negotiation successful
0812 18:53:54.002391 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:53:54.003427 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:53.989001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35535 (local address 127.16.40.193:35143)
0812 18:53:53.989503 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:53.989553 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:53.990934 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:53.991558 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:53.991565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:53.991953 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:53.992482 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:53.992496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.993449 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.993453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:53.994413 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:53.994423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:53.995096 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:53.995101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:53.995291 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:53.995897 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:53.995916 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:53.997898 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:54.000180 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.000186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.000199 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.000484 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.000902 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.000912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.000914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.001082 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.001668 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:54.001674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:54.001936 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 18:53:54.002105 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 18:53:54.002477 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260812 18:53:54.004873 20388 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35535
I20260812 18:53:54.005172 20388 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:54.005843 20388 heartbeater.cc:511] Master 127.16.40.254:35535 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:54.007660 20154 ts_manager.cc:194] Registered new tserver with Master: d352159eda284cc38deb03beb1fce1dd (127.16.40.193:36069)
I20260812 18:53:54.009042 20154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:35143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:54.027709 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:54.145223 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:54.145506 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:54.145586 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:54.149905 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:54.149981 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:54.150077 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:54.156015 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:54.157308 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:54.158433 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:54.168579 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:54.168588 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:54.168588 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:54.168778 20394 server_base.cc:1034] running on GCE node
I20260812 18:53:54.169337 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:54.170094 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:54.171347 20394 hybrid_clock.cc:648] HybridClock initialized: now 1786560834171300 us; error 58 us; skew 500 ppm
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:54.174983 20394 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:54.176117 20394 webserver.cc:460] Webserver started at http://127.16.40.194:35215/ using document root <none> and password file <none>
I20260812 18:53:54.176838 20394 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:54.176913 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:54.177184 20394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:54.179495 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "80d90d33dbaa45a2a24d0f90888dab4f"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "a17de4591b996b4a8ebb490fcfe2637d"
server_key_iv: "b54ebd07dd0e0a3f2fe6c29c24873cdd"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.180136 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "80d90d33dbaa45a2a24d0f90888dab4f"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "a17de4591b996b4a8ebb490fcfe2637d"
server_key_iv: "b54ebd07dd0e0a3f2fe6c29c24873cdd"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.184392 20394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:53:54.187363 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.188895 20394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:54.189061 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "80d90d33dbaa45a2a24d0f90888dab4f"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "a17de4591b996b4a8ebb490fcfe2637d"
server_key_iv: "b54ebd07dd0e0a3f2fe6c29c24873cdd"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.189194 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:54.218434 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:54.222755 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:54.223007 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:54.223807 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:54.224877 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:54.224956 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.225031 20394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:54.225082 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.234923 20394 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:40823
I20260812 18:53:54.234963 20539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:40823 every 8 connection(s)
I20260812 18:53:54.235921 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:54.245769 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20394
I20260812 18:53:54.245916 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:54.246243 16547 external_mini_cluster.cc:1428] Setting key 8b57ce7331b34160a4916325e5c84957
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:54.251842 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:54.238036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:41931 (local address 127.16.40.254:35535)
0812 18:53:54.238187 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:54.238191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:54.239113 (+ 922us) server_negotiation.cc:407] Connection header received
0812 18:53:54.240509 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:54.240512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:54.240646 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:54.240738 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:54.242013 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.242562 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.243789 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.244051 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.244158 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:54.247365 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:54.247369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:54.247372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:54.247413 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:54.249287 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.250038 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.250043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.250046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.250146 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.250551 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.250563 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.250565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.250778 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:54.250903 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:54.251510 (+ 607us) server_negotiation.cc:299] Negotiation successful
0812 18:53:54.251656 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:53:54.252987 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:54.238429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35535 (local address 127.16.40.194:41931)
0812 18:53:54.238919 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:54.238966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:54.240232 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:54.240914 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:54.240921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:54.241329 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:54.241856 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:54.241870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.242691 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.242695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:54.243632 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:54.243642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.244212 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.244215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:54.244364 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:54.245112 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:54.245131 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:54.247161 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:54.249436 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.249443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.249469 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.249907 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.250256 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.250261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.250264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.250437 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.250902 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:54.250923 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:54.251226 (+ 303us) client_negotiation.cc:769] Sending connection context
0812 18:53:54.251452 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 18:53:54.251883 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260812 18:53:54.254433 20540 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35535
I20260812 18:53:54.254763 20540 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:54.255479 20540 heartbeater.cc:511] Master 127.16.40.254:35535 requested a full tablet report, sending...
I20260812 18:53:54.256969 20154 ts_manager.cc:194] Registered new tserver with Master: 80d90d33dbaa45a2a24d0f90888dab4f (127.16.40.194:40823)
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:54.257809 20154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:41931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:54.281235 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:54.399765 20546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:54.400059 20546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:54.400105 20546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:54.405014 20546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:54.405113 20546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:54.405220 20546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:54.411103 20546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:54.412345 20546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:54.413496 20546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:53:54.423620 20546 server_base.cc:1034] running on GCE node
W20260812 18:53:54.423583 20551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:54.423767 20553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:54.423889 20555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:54.424391 20546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:54.425117 20546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:54.426323 20546 hybrid_clock.cc:648] HybridClock initialized: now 1786560834426302 us; error 41 us; skew 500 ppm
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:54.429417 20546 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:54.430577 20546 webserver.cc:460] Webserver started at http://127.16.40.195:34537/ using document root <none> and password file <none>
I20260812 18:53:54.431286 20546 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:54.431358 20546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:54.431598 20546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:54.434391 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "0ccec8301af0450fa5f9e6bbe71bbff9"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "529cdb6db89bf8b15bea132239a55b91"
server_key_iv: "057de951ca197a529bf9059bf87de385"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.435073 20546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "0ccec8301af0450fa5f9e6bbe71bbff9"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "529cdb6db89bf8b15bea132239a55b91"
server_key_iv: "057de951ca197a529bf9059bf87de385"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.439905 20546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 18:53:54.443053 20562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.444497 20546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 18:53:54.444700 20546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "0ccec8301af0450fa5f9e6bbe71bbff9"
format_stamp: "Formatted at 2026-08-12 18:53:54 on dist-test-slave-rnrw"
server_key: "529cdb6db89bf8b15bea132239a55b91"
server_key_iv: "057de951ca197a529bf9059bf87de385"
server_key_version: "encryptionkey@0"
I20260812 18:53:54.444828 20546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:54.459904 20546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:54.463639 20546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:54.463855 20546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:54.464499 20546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:54.465448 20546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:54.465497 20546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.465569 20546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:54.465608 20546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:54.477157 20546 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39535
I20260812 18:53:54.477283 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39535 every 8 connection(s)
I20260812 18:53:54.478188 20546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:54.479281 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20546
I20260812 18:53:54.479426 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:54.479779 16547 external_mini_cluster.cc:1428] Setting key 78b6f14792b1d29b71c03908138f71bb
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:54.495077 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:54.480571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:47987 (local address 127.16.40.254:35535)
0812 18:53:54.480863 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:54.480868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:54.482142 (+ 1274us) server_negotiation.cc:407] Connection header received
0812 18:53:54.483264 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:54.483268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:54.483323 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:54.483414 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:54.484826 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.485735 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.486942 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.487186 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.487307 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:54.490236 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:54.490243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:54.490247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:54.490283 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:54.492031 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.492662 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.492666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.492667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.492753 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.493475 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.493478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.493480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.493717 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:54.493841 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:54.494555 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 18:53:54.494722 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260812 18:53:54.495880 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:54.480948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35535 (local address 127.16.40.195:47987)
0812 18:53:54.481519 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:54.481584 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:54.483053 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:54.483622 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:54.483630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:54.484020 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:54.484663 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:54.484685 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.485870 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.485873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:54.486774 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:54.486784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.487379 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.487384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:54.487571 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:54.488352 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:54.488371 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:54.490028 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:54.492162 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.492168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.492183 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.492473 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.492858 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:54.492864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:54.492868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:54.493075 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:54.493842 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:54.493851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:54.494210 (+ 359us) client_negotiation.cc:769] Sending connection context
0812 18:53:54.494494 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 18:53:54.494869 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":113,"threads_started":1}
I20260812 18:53:54.497282 20694 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35535
I20260812 18:53:54.497596 20694 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:54.498309 20694 heartbeater.cc:511] Master 127.16.40.254:35535 requested a full tablet report, sending...
I20260812 18:53:54.499558 20154 ts_manager.cc:194] Registered new tserver with Master: 0ccec8301af0450fa5f9e6bbe71bbff9 (127.16.40.195:39535)
I20260812 18:53:54.500211 20154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:47987
I20260812 18:53:54.506758 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560834, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:54.543313 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:54.530472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.16.40.254:35535)
0812 18:53:54.530606 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:54.530611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:54.532301 (+ 1690us) server_negotiation.cc:407] Connection header received
0812 18:53:54.532529 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:54.532533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:54.532626 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:54.532686 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:54.533964 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.534982 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.536315 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:54.536835 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:54.536943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:54.539661 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:54.539665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:54.539667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:54.539697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:54.541351 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.541853 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.541857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.541858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.541960 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:54.542310 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:54.542313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:54.542314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:54.542653 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:54.542771 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:54.543062 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 18:53:54.543181 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 18:53:54.543910 20154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53960
I20260812 18:53:54.545063 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20229
I20260812 18:53:54.555290 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20394
I20260812 18:53:54.568336 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20546
I20260812 18:53:54.578531 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20115
2026-08-12T18:53:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2424 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20709](info): set up 2 sockets
Aug 12 18:53:54 dist-test-slave-rnrw krb5kdc[20709](info): commencing operation
krb5kdc: starting...
W20260812 18:53:55.716598 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:55 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:55.863962 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35395
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:55.985884 20725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:55.986220 20725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:55.986268 20725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:55.990718 20725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:55.990814 20725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:55.990834 20725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:55.990851 20725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:55.990867 20725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:55.996802 20725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:55.998085 20725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:55.999239 20725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:56.008729 20730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.008738 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.008754 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:56.008895 20725 server_base.cc:1034] running on GCE node
I20260812 18:53:56.009848 20725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:56.011144 20725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:56.012372 20725 hybrid_clock.cc:648] HybridClock initialized: now 1786560836012331 us; error 54 us; skew 500 ppm
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:56.015776 20725 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:56.017030 20725 webserver.cc:460] Webserver started at http://127.16.40.254:40999/ using document root <none> and password file <none>
I20260812 18:53:56.017675 20725 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:56.017779 20725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:56.018038 20725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:56.020325 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "541e2aa0449e43a58c576fbfe07ab867"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "493a9c4e7e59bca2cd0460f159187086"
server_key_iv: "b3e431ee631167ea2da34765eb8ed3b5"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.021050 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "541e2aa0449e43a58c576fbfe07ab867"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "493a9c4e7e59bca2cd0460f159187086"
server_key_iv: "b3e431ee631167ea2da34765eb8ed3b5"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.025568 20725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:56.028452 20742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.029992 20725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:56.030194 20725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "541e2aa0449e43a58c576fbfe07ab867"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "493a9c4e7e59bca2cd0460f159187086"
server_key_iv: "b3e431ee631167ea2da34765eb8ed3b5"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.030351 20725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:56.059154 20725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:56.062709 20725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:56.062942 20725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:56.070559 20725 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:35395
I20260812 18:53:56.070573 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:35395 every 8 connection(s)
I20260812 18:53:56.071684 20725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:56.072744 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20725
I20260812 18:53:56.072849 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:56.073163 16547 external_mini_cluster.cc:1428] Setting key 6310b66454739688e72e4adb73325aac
I20260812 18:53:56.075512 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:56.083335 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867: Bootstrap starting.
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:56.086499 20802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:56.087584 20802 log.cc:824] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:56.089942 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.074751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.16.40.254:35395)
0812 18:53:56.075217 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.075238 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.075286 (+ 48us) server_negotiation.cc:407] Connection header received
0812 18:53:56.076349 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.076376 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.076879 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.077295 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.078059 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.079607 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.080500 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.081240 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.081429 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.084117 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.084125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.084144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.084183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.086404 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.087039 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.087046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.087053 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.087219 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.087595 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.087601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.087605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.088211 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.088432 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.088824 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.089159 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":120,"threads_started":1}
I20260812 18:53:56.089989 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867: No bootstrap required, opened a new log
I20260812 18:53:56.094098 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } }
I20260812 18:53:56.094429 20802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:56.094521 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541e2aa0449e43a58c576fbfe07ab867, State: Initialized, Role: FOLLOWER
I20260812 18:53:56.095032 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } }
I20260812 18:53:56.095207 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:56.095302 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:56.095429 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:56.096503 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } }
I20260812 18:53:56.096930 20802 leader_election.cc:302] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541e2aa0449e43a58c576fbfe07ab867; no voters:
I20260812 18:53:56.097334 20802 leader_election.cc:288] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:56.097642 20807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:56.097934 20807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [term 1 LEADER]: Becoming Leader. State: Replica: 541e2aa0449e43a58c576fbfe07ab867, State: Running, Role: LEADER
I20260812 18:53:56.098280 20807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } }
I20260812 18:53:56.098881 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:56.100250 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541e2aa0449e43a58c576fbfe07ab867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } } }
I20260812 18:53:56.100315 20809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541e2aa0449e43a58c576fbfe07ab867. Latest consensus state: current_term: 1 leader_uuid: "541e2aa0449e43a58c576fbfe07ab867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541e2aa0449e43a58c576fbfe07ab867" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35395 } } }
I20260812 18:53:56.100391 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:56.100435 20809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:56.100795 20818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:56.103849 20818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:56.110091 20818 catalog_manager.cc:1347] Generated new cluster ID: cbbb58d0f99c43268732ba4035dab501
I20260812 18:53:56.110190 20818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:56.124740 20818 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:56.125808 20818 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:56.133875 20818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541e2aa0449e43a58c576fbfe07ab867: Generated new TSK 0
I20260812 18:53:56.134799 20818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:56.172369 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:56.294685 20830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:56.294978 20830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:56.295027 20830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:56.299185 20830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:56.299292 20830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:56.299381 20830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:56.304868 20830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:56.305986 20830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:56.306996 20830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:56.317387 20836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.317402 20838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.317387 20835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:56.317659 20830 server_base.cc:1034] running on GCE node
I20260812 18:53:56.318246 20830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:56.319000 20830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:56.320214 20830 hybrid_clock.cc:648] HybridClock initialized: now 1786560836320197 us; error 28 us; skew 500 ppm
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:56.323448 20830 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:56.324586 20830 webserver.cc:460] Webserver started at http://127.16.40.193:37899/ using document root <none> and password file <none>
I20260812 18:53:56.325260 20830 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:56.325366 20830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:56.325637 20830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:56.327986 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "91d0481525f8417792fe753adb583968"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "ac9eaf4062343fd5b793204bf6645db3"
server_key_iv: "03959718fce23dfb8d1510c0099eea83"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.328727 20830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "91d0481525f8417792fe753adb583968"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "ac9eaf4062343fd5b793204bf6645db3"
server_key_iv: "03959718fce23dfb8d1510c0099eea83"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.333140 20830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:56.336382 20847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.338008 20830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:56.338188 20830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "91d0481525f8417792fe753adb583968"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "ac9eaf4062343fd5b793204bf6645db3"
server_key_iv: "03959718fce23dfb8d1510c0099eea83"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.338338 20830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:56.355279 20830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:56.359179 20830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:56.359432 20830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:56.360180 20830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:56.361188 20830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:56.361266 20830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.361346 20830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:56.361389 20830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.371642 20830 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:46481
I20260812 18:53:56.371685 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:46481 every 8 connection(s)
I20260812 18:53:56.372692 20830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:56.381039 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20830
I20260812 18:53:56.381191 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:56.381539 16547 external_mini_cluster.cc:1428] Setting key 86b4856a481e15ff9db90a61dc4e7799
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:56.388510 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.374966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:33429 (local address 127.16.40.254:35395)
0812 18:53:56.375173 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.375177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.376128 (+ 951us) server_negotiation.cc:407] Connection header received
0812 18:53:56.377764 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.377767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.377822 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.377907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.379267 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.379818 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.381323 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.381568 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.381701 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.384605 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.384609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.384611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.384634 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.386245 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.386823 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.386827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.386829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.386917 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.387288 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.387293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.387295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.387524 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.387639 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.388179 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.388322 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 18:53:56.389441 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.375397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35395 (local address 127.16.40.193:33429)
0812 18:53:56.375949 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:56.375997 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:56.377495 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:56.378107 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:56.378114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:56.378576 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:56.379104 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.379118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.379946 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.379949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.381125 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.381137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.381753 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.381757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.381967 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.382700 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:56.382721 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:56.384402 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:56.386395 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.386403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.386422 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.386713 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.387021 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.387024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.387026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.387184 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.387639 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:56.387647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:56.387926 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 18:53:56.388139 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 18:53:56.388507 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":128,"threads_started":1}
I20260812 18:53:56.390834 20981 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35395
I20260812 18:53:56.391170 20981 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:56.391924 20981 heartbeater.cc:511] Master 127.16.40.254:35395 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:56.393894 20762 ts_manager.cc:194] Registered new tserver with Master: 91d0481525f8417792fe753adb583968 (127.16.40.193:46481)
I20260812 18:53:56.395165 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:33429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:56.416643 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:56.545710 20986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:56.546051 20986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:56.546099 20986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:56.550590 20986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:56.550676 20986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:56.550740 20986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:56.556723 20986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:56.557937 20986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:56.559031 20986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:56.569599 20994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.569545 20992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.569600 20991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:56.569921 20986 server_base.cc:1034] running on GCE node
I20260812 18:53:56.570348 20986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:56.571112 20986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:56.572304 20986 hybrid_clock.cc:648] HybridClock initialized: now 1786560836572263 us; error 54 us; skew 500 ppm
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:56.575659 20986 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:56.576804 20986 webserver.cc:460] Webserver started at http://127.16.40.194:37681/ using document root <none> and password file <none>
I20260812 18:53:56.577451 20986 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:56.577556 20986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:56.577817 20986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:56.580065 20986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "fff07b80d9ae4476886c50c1a1cc9015"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "15a40e4b730d5f9563788757cf8ab71f"
server_key_iv: "6d64caf996aebff80aa0507fbc7e38e4"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.580745 20986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "fff07b80d9ae4476886c50c1a1cc9015"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "15a40e4b730d5f9563788757cf8ab71f"
server_key_iv: "6d64caf996aebff80aa0507fbc7e38e4"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.584970 20986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:56.588367 21002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.589903 20986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:53:56.590161 20986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "fff07b80d9ae4476886c50c1a1cc9015"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "15a40e4b730d5f9563788757cf8ab71f"
server_key_iv: "6d64caf996aebff80aa0507fbc7e38e4"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.590299 20986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:56.636123 20986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:56.639750 20986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:56.640017 20986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:56.640777 20986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:56.641853 20986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:56.641932 20986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.642006 20986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:56.642084 20986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.652873 20986 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33893
I20260812 18:53:56.652894 21135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33893 every 8 connection(s)
I20260812 18:53:56.653919 20986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:56.656826 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 20986
I20260812 18:53:56.656962 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:56.657290 16547 external_mini_cluster.cc:1428] Setting key 3f8e2461592775bf4952ad7de5a09d35
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:56.671209 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.656407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:44709 (local address 127.16.40.254:35395)
0812 18:53:56.656675 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.656681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.657782 (+ 1101us) server_negotiation.cc:407] Connection header received
0812 18:53:56.659610 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.659613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.659661 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.659735 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.661650 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.662547 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.663840 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.664099 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.664227 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.667032 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.667036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.667038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.667065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.668826 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.669423 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.669426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.669428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.669535 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.669967 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.669969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.669971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.670171 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.670251 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.670950 (+ 699us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.671075 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:53:56.672312 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.656919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35395 (local address 127.16.40.194:44709)
0812 18:53:56.657576 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:56.657644 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:56.659333 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:56.659969 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:56.659983 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:56.660610 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:56.661422 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.661449 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.662714 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.662717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.663693 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.663704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.664280 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.664284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.664471 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.665254 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:56.665274 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:56.666812 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:56.668976 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.668984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.669003 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.669299 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.669668 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.669671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.669672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.669834 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.670270 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:56.670276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:56.670623 (+ 347us) client_negotiation.cc:769] Sending connection context
0812 18:53:56.670843 (+ 220us) client_negotiation.cc:240] Negotiation successful
0812 18:53:56.671151 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":130,"threads_started":1}
I20260812 18:53:56.673688 21136 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35395
I20260812 18:53:56.673960 21136 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:56.674502 21136 heartbeater.cc:511] Master 127.16.40.254:35395 requested a full tablet report, sending...
I20260812 18:53:56.675580 20762 ts_manager.cc:194] Registered new tserver with Master: fff07b80d9ae4476886c50c1a1cc9015 (127.16.40.194:33893)
I20260812 18:53:56.676188 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:44709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:56.689862 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:56.813319 21141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:56.813632 21141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:56.813678 21141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:56.817915 21141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:56.817994 21141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:56.818058 21141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:56.823686 21141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:56.824913 21141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:56.825933 21141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:56.836258 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.836258 21152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:56.836316 21149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:56.836350 21141 server_base.cc:1034] running on GCE node
I20260812 18:53:56.837023 21141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:56.837772 21141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:56.838972 21141 hybrid_clock.cc:648] HybridClock initialized: now 1786560836838955 us; error 29 us; skew 500 ppm
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:56.842047 21141 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:56.843106 21141 webserver.cc:460] Webserver started at http://127.16.40.195:37951/ using document root <none> and password file <none>
I20260812 18:53:56.843734 21141 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:56.843830 21141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:56.844063 21141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:56.846318 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "c8e30c34acc54ec3889876314700539b"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "c126e9dc64b198b0eb849516c005cbac"
server_key_iv: "83452d303a3883c5f3f23ea28d694707"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.846909 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "c8e30c34acc54ec3889876314700539b"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "c126e9dc64b198b0eb849516c005cbac"
server_key_iv: "83452d303a3883c5f3f23ea28d694707"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.851107 21141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:56.853857 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.855234 21141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:53:56.855398 21141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "c8e30c34acc54ec3889876314700539b"
format_stamp: "Formatted at 2026-08-12 18:53:56 on dist-test-slave-rnrw"
server_key: "c126e9dc64b198b0eb849516c005cbac"
server_key_iv: "83452d303a3883c5f3f23ea28d694707"
server_key_version: "encryptionkey@0"
I20260812 18:53:56.855533 21141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:56.866596 21141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:56.869992 21141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:56.870210 21141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:56.870862 21141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:56.871773 21141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:56.871822 21141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.871892 21141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:56.871929 21141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:56.882033 21141 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:46253
I20260812 18:53:56.882053 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:46253 every 8 connection(s)
I20260812 18:53:56.883015 21141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:56.888274 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21141
I20260812 18:53:56.888445 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:56.888808 16547 external_mini_cluster.cc:1428] Setting key eb0cc3f64e9bb29ac1aebf3cea2fe186
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:56.898475 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.885109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:37391 (local address 127.16.40.254:35395)
0812 18:53:56.885275 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.885280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.886204 (+ 924us) server_negotiation.cc:407] Connection header received
0812 18:53:56.887637 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.887640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.887692 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.887775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.889593 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.890322 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.891782 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.891996 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.892080 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.894798 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.894801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.894803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.894828 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.896371 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.896902 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.896905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.896907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.897010 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.897401 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.897405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.897406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.897609 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.897701 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.898228 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.898343 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:53:56.899410 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.885524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35395 (local address 127.16.40.195:37391)
0812 18:53:56.886036 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:56.886084 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:56.887368 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:56.888026 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:56.888038 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:56.888602 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:56.889330 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.889355 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.890467 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.890470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.891663 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:56.891673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.892162 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.892166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:56.892330 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.893046 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:56.893063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:56.894635 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:56.896490 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.896496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.896510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.896804 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.897095 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:56.897098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:56.897099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:56.897272 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:56.897713 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:56.897719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:56.897968 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 18:53:56.898136 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 18:53:56.898407 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":110,"threads_started":1}
I20260812 18:53:56.900806 21297 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35395
I20260812 18:53:56.901051 21297 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:56.901552 21297 heartbeater.cc:511] Master 127.16.40.254:35395 requested a full tablet report, sending...
I20260812 18:53:56.902761 20762 ts_manager.cc:194] Registered new tserver with Master: c8e30c34acc54ec3889876314700539b (127.16.40.195:46253)
I20260812 18:53:56.903354 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:53:56.903400 20762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:37391
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:53:56.932487 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.922675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.16.40.193:46481)
0812 18:53:56.923013 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.923022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.923051 (+ 29us) server_negotiation.cc:407] Connection header received
0812 18:53:56.923112 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.923118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.923313 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.923440 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.924149 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.925051 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.925977 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.926337 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.926420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.929013 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.929025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.929041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.929066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.930703 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.931127 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.931132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.931133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.931212 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.931468 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.931470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.931472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.931751 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.931911 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.932119 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.932293 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":158,"threads_started":1}
Aug 12 18:53:56 dist-test-slave-rnrw krb5kdc[20709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:56.944393 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:56.935703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.16.40.254:35395)
0812 18:53:56.935826 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:56.935830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:56.935878 (+ 48us) server_negotiation.cc:407] Connection header received
0812 18:53:56.936002 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:56.936004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:56.936051 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:56.936126 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:56.936854 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.937382 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.938215 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:56.938502 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:56.938606 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:56.941019 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:56.941023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:56.941025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:56.941053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:56.942603 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.943001 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.943004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.943006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.943088 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:56.943341 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:56.943343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:56.943344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:56.943635 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:56.943746 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:56.944095 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 18:53:56.944260 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 18:53:56.944929 20762 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52672
I20260812 18:53:56.945703 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20830
I20260812 18:53:56.958091 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20986
I20260812 18:53:56.971127 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21141
I20260812 18:53:56.984845 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 20725
2026-08-12T18:53:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2405 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:53:57 dist-test-slave-rnrw krb5kdc[21313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:53:57 dist-test-slave-rnrw krb5kdc[21313](info): set up 2 sockets
Aug 12 18:53:57 dist-test-slave-rnrw krb5kdc[21313](info): commencing operation
krb5kdc: starting...
W20260812 18:53:58.121415 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:53:58.260310 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33283
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:58.382643 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:58.382941 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:58.382985 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:58.387108 21328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:58.387187 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:58.387207 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:58.387224 21328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:58.387239 21328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:58.392884 21328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:58.394119 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:58.395248 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:58.404300 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:58.404460 21328 server_base.cc:1034] running on GCE node
W20260812 18:53:58.404322 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:58.404300 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:58.405241 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:58.406517 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:58.407770 21328 hybrid_clock.cc:648] HybridClock initialized: now 1786560838407731 us; error 43 us; skew 500 ppm
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:58.410981 21328 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:53:58.411823 21328 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:58.411931 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:58.412205 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:58.414459 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "a9e4eacb45aac569aa3f985b90773754"
server_key_iv: "99077399f54d29cc37830b6bd4e9d6e8"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.415071 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "a9e4eacb45aac569aa3f985b90773754"
server_key_iv: "99077399f54d29cc37830b6bd4e9d6e8"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.419090 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:53:58.421788 21346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.423162 21328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:53:58.423326 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "a9e4eacb45aac569aa3f985b90773754"
server_key_iv: "99077399f54d29cc37830b6bd4e9d6e8"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.423454 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:58.435235 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:58.438953 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:58.439182 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:58.446703 21328 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33283
I20260812 18:53:58.447043 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33283 every 8 connection(s)
I20260812 18:53:58.447528 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:58.448629 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21328
I20260812 18:53:58.448769 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:58.449052 16547 external_mini_cluster.cc:1428] Setting key 83cec0e16f80ef438015b271ba5d1d7e
I20260812 18:53:58.451366 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:58.458954 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: Bootstrap starting.
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:58.462051 21410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:58.463204 21410 log.cc:824] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:58.466246 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: No bootstrap required, opened a new log
I20260812 18:53:58.466852 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:58.450658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.16.40.254:33283)
0812 18:53:58.451140 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:58.451160 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:58.451207 (+ 47us) server_negotiation.cc:407] Connection header received
0812 18:53:58.452331 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:58.452355 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:58.452824 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:58.453228 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:58.453949 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.455527 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.456693 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.457432 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.457612 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:58.460662 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:58.460669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:58.460689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:58.460729 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:58.462912 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:58.463631 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:58.463640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:58.463648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:58.463825 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:58.464201 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:58.464204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:58.464206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:58.464872 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:58.465126 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:58.465509 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 18:53:58.465842 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260812 18:53:58.470644 21410 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } }
I20260812 18:53:58.470979 21410 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:58.471061 21410 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e8f19c86a04770b4bc1cd0c0da08f1, State: Initialized, Role: FOLLOWER
I20260812 18:53:58.471590 21410 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } }
I20260812 18:53:58.471805 21410 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:58.471908 21410 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:58.472059 21410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:58.473532 21410 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } }
I20260812 18:53:58.474009 21410 leader_election.cc:302] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e8f19c86a04770b4bc1cd0c0da08f1; no voters:
I20260812 18:53:58.474501 21410 leader_election.cc:288] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:58.474741 21417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:58.475088 21417 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [term 1 LEADER]: Becoming Leader. State: Replica: e6e8f19c86a04770b4bc1cd0c0da08f1, State: Running, Role: LEADER
I20260812 18:53:58.475482 21417 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } }
I20260812 18:53:58.475903 21410 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:58.477483 21418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } } }
I20260812 18:53:58.477519 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e8f19c86a04770b4bc1cd0c0da08f1. Latest consensus state: current_term: 1 leader_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e8f19c86a04770b4bc1cd0c0da08f1" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33283 } } }
I20260812 18:53:58.477675 21418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:58.477747 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1 [sys.catalog]: This master's current role is: LEADER
W20260812 18:53:58.481925 21435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:53:58.482010 21435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:53:58.482105 21430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:58.482913 21430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:58.489203 21430 catalog_manager.cc:1347] Generated new cluster ID: 09e105c2dd3c4724bdf8334317cbd0cd
I20260812 18:53:58.489310 21430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:58.516106 21430 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:58.517336 21430 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:58.533977 21430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: Generated new TSK 0
I20260812 18:53:58.534901 21430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:53:58.571797 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:58.695190 21442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:58.695498 21442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:58.695545 21442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:58.699788 21442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:58.699865 21442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:58.699929 21442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:58.705432 21442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:58.706570 21442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:58.707613 21442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:58.717989 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:58.717971 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:58.717989 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:58.718444 21442 server_base.cc:1034] running on GCE node
I20260812 18:53:58.718892 21442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:58.719635 21442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:58.720829 21442 hybrid_clock.cc:648] HybridClock initialized: now 1786560838720805 us; error 33 us; skew 500 ppm
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:58.723875 21442 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:53:58.724748 21442 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:58.724846 21442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:58.725102 21442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:58.727545 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "67718504b81c47628ea093b47fc8c015"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "f78bbe13bbe5b7a6012dcf80230da462"
server_key_iv: "71d2c2455b0ea8d0cc9af2ad8032db13"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.728148 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "67718504b81c47628ea093b47fc8c015"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "f78bbe13bbe5b7a6012dcf80230da462"
server_key_iv: "71d2c2455b0ea8d0cc9af2ad8032db13"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.731956 21442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 18:53:58.734532 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.735764 21442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 18:53:58.735916 21442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "67718504b81c47628ea093b47fc8c015"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "f78bbe13bbe5b7a6012dcf80230da462"
server_key_iv: "71d2c2455b0ea8d0cc9af2ad8032db13"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.736044 21442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:58.750689 21442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:58.753877 21442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:58.754106 21442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:58.754751 21442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:58.755676 21442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:58.755751 21442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.755828 21442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:58.755882 21442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.765560 21442 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:40991
I20260812 18:53:58.765573 21596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:40991 every 8 connection(s)
I20260812 18:53:58.766237 21442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:58.770176 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21442
I20260812 18:53:58.770331 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:58.770663 16547 external_mini_cluster.cc:1428] Setting key dda1943991cf9d8c2b07e5aa09278e48
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:53:58.783692 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:58.768257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:50959 (local address 127.16.40.254:33283)
0812 18:53:58.768446 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:58.768452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:58.769332 (+ 880us) server_negotiation.cc:407] Connection header received
0812 18:53:58.770692 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:58.770695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:58.770759 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:58.770872 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:58.772247 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.773055 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.774279 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.774509 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.774646 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:58.778075 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:58.778083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:58.778089 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:58.778152 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:58.780689 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:58.781437 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:58.781442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:58.781444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:58.781564 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:58.782058 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:58.782061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:58.782063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:58.782337 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:58.782453 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:58.783286 (+ 833us) server_negotiation.cc:299] Negotiation successful
0812 18:53:58.783497 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:53:58.785014 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:58.768640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33283 (local address 127.16.40.193:50959)
0812 18:53:58.769169 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:58.769216 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:58.770452 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:58.771098 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:58.771107 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:58.771531 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:58.772085 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:58.772105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.773195 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.773198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:58.774141 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:58.774151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:58.774704 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:58.774708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:58.774888 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:58.775597 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:58.775615 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:58.777829 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:58.780858 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:58.780866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:58.780886 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:58.781294 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:58.781704 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:58.781707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:58.781709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:58.781931 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:58.782475 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:58.782485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:58.782975 (+ 490us) client_negotiation.cc:769] Sending connection context
0812 18:53:58.783234 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 18:53:58.783696 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:53:58.786808 21598 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33283
I20260812 18:53:58.787102 21598 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:58.787635 21598 heartbeater.cc:511] Master 127.16.40.254:33283 requested a full tablet report, sending...
I20260812 18:53:58.789395 21368 ts_manager.cc:194] Registered new tserver with Master: 67718504b81c47628ea093b47fc8c015 (127.16.40.193:40991)
I20260812 18:53:58.791147 21368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:50959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:53:58.804634 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:58.924523 21603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:58.924891 21603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:58.924947 21603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:58.929142 21603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:58.929211 21603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:58.929275 21603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:53:58.934809 21603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:58.935904 21603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:58.937034 21603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:58.947268 21609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:58.947263 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:58.947263 21608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:58.947710 21603 server_base.cc:1034] running on GCE node
I20260812 18:53:58.948100 21603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:58.948839 21603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:58.950032 21603 hybrid_clock.cc:648] HybridClock initialized: now 1786560838949995 us; error 49 us; skew 500 ppm
Aug 12 18:53:58 dist-test-slave-rnrw krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:58.953058 21603 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:53:58.953948 21603 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:58.954059 21603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:58.954320 21603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:58.956660 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "21a50671d2ef4da193d3a5f91b2868a4"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "d3058e481e1be926c8b4efd7a565ebcc"
server_key_iv: "a9bda55abbeb48c31ab37c01b4326c96"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.957290 21603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "21a50671d2ef4da193d3a5f91b2868a4"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "d3058e481e1be926c8b4efd7a565ebcc"
server_key_iv: "a9bda55abbeb48c31ab37c01b4326c96"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.961409 21603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 18:53:58.964190 21618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.965612 21603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:53:58.965777 21603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "21a50671d2ef4da193d3a5f91b2868a4"
format_stamp: "Formatted at 2026-08-12 18:53:58 on dist-test-slave-rnrw"
server_key: "d3058e481e1be926c8b4efd7a565ebcc"
server_key_iv: "a9bda55abbeb48c31ab37c01b4326c96"
server_key_version: "encryptionkey@0"
I20260812 18:53:58.965904 21603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:58.979552 21603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:58.982861 21603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:58.983098 21603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:58.983722 21603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:58.984696 21603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:58.984768 21603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.984843 21603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:58.984884 21603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:58.994333 21603 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:37845
I20260812 18:53:58.994352 21746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:37845 every 8 connection(s)
I20260812 18:53:58.994995 21603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:53:59.002539 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21603
I20260812 18:53:59.002696 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:53:59.002995 16547 external_mini_cluster.cc:1428] Setting key f92fa4623431c30ce29ec5fd8f4fc1e6
Aug 12 18:53:59 dist-test-slave-rnrw krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:59.010480 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:58.996930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:37467 (local address 127.16.40.254:33283)
0812 18:53:58.997071 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:58.997080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:58.997953 (+ 873us) server_negotiation.cc:407] Connection header received
0812 18:53:58.999274 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:58.999277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:58.999341 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:58.999450 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:59.000750 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.001426 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.002628 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.002974 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.003108 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.006023 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:59.006026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:59.006028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:59.006053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:59.007898 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.008487 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.008493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.008498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.008626 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.008984 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.008990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.008994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.009277 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:59.009392 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.010045 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.010218 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 18:53:59.011284 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:58.997287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33283 (local address 127.16.40.194:37467)
0812 18:53:58.997788 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:58.997836 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:58.999043 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:58.999642 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:58.999649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.000015 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:59.000519 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.000533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.001563 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.001567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.002489 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.002498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.003146 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.003150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.003341 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.004062 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:59.004084 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:59.005838 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:59.008063 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:59.008069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:59.008085 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:59.008371 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:59.008731 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:59.008734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:59.008735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:59.008882 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:59.009434 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:59.009440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:59.009711 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.009896 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.010265 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":123,"threads_started":1}
I20260812 18:53:59.012758 21748 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33283
I20260812 18:53:59.013058 21748 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:59.013695 21748 heartbeater.cc:511] Master 127.16.40.254:33283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:53:59.015076 21368 ts_manager.cc:194] Registered new tserver with Master: 21a50671d2ef4da193d3a5f91b2868a4 (127.16.40.194:37845)
I20260812 18:53:59.015710 21368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:37467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:53:59.034204 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:53:59.151199 21753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:59.151602 21753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:59.151700 21753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:59.155843 21753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:59.155948 21753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:59.156055 21753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:53:59.161784 21753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:59.162973 21753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:59.164062 21753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:59.173918 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:59.173918 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:59.173941 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:59.174278 21753 server_base.cc:1034] running on GCE node
I20260812 18:53:59.174741 21753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:59.175447 21753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:59.176648 21753 hybrid_clock.cc:648] HybridClock initialized: now 1786560839176630 us; error 35 us; skew 500 ppm
Aug 12 18:53:59 dist-test-slave-rnrw krb5kdc[21313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:53:59.179529 21753 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:53:59.180296 21753 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:59.180394 21753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:59.180732 21753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:59.182914 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "1f6385d2429649428cc2f06e1eaafdc3"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "91f0e91a5097d7215d0c948e3926de3f"
server_key_iv: "609a52d1daa3f2a26ab41a2bcfb09703"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.183521 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "1f6385d2429649428cc2f06e1eaafdc3"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "91f0e91a5097d7215d0c948e3926de3f"
server_key_iv: "609a52d1daa3f2a26ab41a2bcfb09703"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.187445 21753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:53:59.190040 21769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.191287 21753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 18:53:59.191433 21753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "1f6385d2429649428cc2f06e1eaafdc3"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "91f0e91a5097d7215d0c948e3926de3f"
server_key_iv: "609a52d1daa3f2a26ab41a2bcfb09703"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.191552 21753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:59.206323 21753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:59.209619 21753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:59.209780 21753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:59.210347 21753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:59.211179 21753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:59.211228 21753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.211269 21753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:59.211285 21753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.220427 21753 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:44671
I20260812 18:53:59.220445 21907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:44671 every 8 connection(s)
I20260812 18:53:59.221170 21753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:53:59.222100 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21753
I20260812 18:53:59.222261 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:53:59.222566 16547 external_mini_cluster.cc:1428] Setting key bbdac3307abdfd0b7726bea4130cf415
Aug 12 18:53:59 dist-test-slave-rnrw krb5kdc[21313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:53:59.235498 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.223246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:52193 (local address 127.16.40.254:33283)
0812 18:53:59.223385 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.223389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.224495 (+ 1106us) server_negotiation.cc:407] Connection header received
0812 18:53:59.225887 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.225889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.225936 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.226002 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:59.227157 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.227740 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.228886 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.229097 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.229185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.231934 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:59.231937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:59.231940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:59.231965 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:59.233383 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.233907 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.233909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.233910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.234004 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.234385 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.234388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.234389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.234580 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:59.234658 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.235197 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.235314 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:53:59.236521 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.223777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33283 (local address 127.16.40.195:52193)
0812 18:53:59.224323 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.224378 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.225687 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.226143 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.226150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.226510 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:59.227029 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.227043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.227838 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.227841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.228780 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.228790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.229237 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.229240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.229378 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.229944 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:53:59.229960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:59.231744 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:59.233498 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:59.233503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:59.233540 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:59.233813 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:59.234118 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:53:59.234125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:53:59.234130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:53:59.234295 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:53:59.234673 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:59.234678 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:59.234926 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.235152 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.235507 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":162,"threads_started":1}
I20260812 18:53:59.237812 21908 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33283
I20260812 18:53:59.238080 21908 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:59.238569 21908 heartbeater.cc:511] Master 127.16.40.254:33283 requested a full tablet report, sending...
I20260812 18:53:59.239693 21368 ts_manager.cc:194] Registered new tserver with Master: 1f6385d2429649428cc2f06e1eaafdc3 (127.16.40.195:44671)
I20260812 18:53:59.240278 21368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:52193
I20260812 18:53:59.249581 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:53:59.259963 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.251908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.16.40.254:33283)
0812 18:53:59.252061 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.252066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.252212 (+ 146us) server_negotiation.cc:407] Connection header received
0812 18:53:59.252320 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.252323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.252364 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.252453 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:59.253263 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.253818 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.254776 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.255032 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.255128 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.256787 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:59.256791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:59.256792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:53:59.256821 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:59.258371 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.258808 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.258811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.258812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.258893 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:53:59.259148 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:53:59.259150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:53:59.259151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:53:59.259382 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:59.259491 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.259728 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.259832 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:53:59.263335 21368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:53:59.266667 21368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:53:59.283713 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.277647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.16.40.194:37845)
0812 18:53:59.277993 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.278002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.278022 (+ 20us) server_negotiation.cc:407] Connection header received
0812 18:53:59.278084 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.278088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.278319 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.278479 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:59.278822 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.280127 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.282018 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.282936 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.283073 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.283254 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.283421 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.283530 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":138,"threads_started":1}
I20260812 18:53:59.283708 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.277389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37845 (local address 127.0.0.1:35706)
0812 18:53:59.277688 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.277700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.277825 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.278474 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.278481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.278506 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:59.278702 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.278709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.280278 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.280281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.281869 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.281881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.283072 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.283075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.283190 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.283230 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.283385 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.283547 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":46,"thread_start_us":110,"threads_started":1}
I20260812 18:53:59.284473 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.276296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:44671 (local address 127.0.0.1:58238)
0812 18:53:59.277114 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.277172 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.277315 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.278545 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.278548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.278570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:59.278810 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.278818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.281615 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.281619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.282754 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.282766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.284043 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.284050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.284158 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.284188 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.284252 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.284325 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":86,"thread_start_us":101,"threads_started":1}
I20260812 18:53:59.284479 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.276386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.16.40.193:40991)
0812 18:53:59.277452 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.277462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.277780 (+ 318us) server_negotiation.cc:407] Connection header received
0812 18:53:59.277959 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.277966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.278194 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.278354 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:59.278944 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.280103 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.282569 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.283349 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.283460 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.283619 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.284228 (+ 609us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.284335 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":91,"threads_started":1}
I20260812 18:53:59.284941 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.276296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:40991 (local address 127.0.0.1:49556)
0812 18:53:59.277647 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.277660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.277773 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.278579 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.278582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.278597 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:59.278816 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.278822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.281341 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.281347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.282420 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.282429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.283692 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.283696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.283803 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.284082 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.284728 (+ 646us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.284815 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":826,"thread_start_us":113,"threads_started":1}
I20260812 18:53:59.285000 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.276652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.16.40.195:44671)
0812 18:53:59.277300 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.277318 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.277348 (+ 30us) server_negotiation.cc:407] Connection header received
0812 18:53:59.277996 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.278002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.278194 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.278354 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:59.280285 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.281461 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.282899 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.283724 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.283828 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.283962 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.284753 (+ 791us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.284854 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":171,"threads_started":1}
I20260812 18:53:59.287503 21836 tablet_service.cc:1467] Processing CreateTablet for tablet f8993feb727a4e0ebbf083bba3657561 (DEFAULT_TABLE table=test-table [id=eb126534565e453ab51e74dafa595992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:59.287570 21673 tablet_service.cc:1467] Processing CreateTablet for tablet f8993feb727a4e0ebbf083bba3657561 (DEFAULT_TABLE table=test-table [id=eb126534565e453ab51e74dafa595992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:59.287726 21519 tablet_service.cc:1467] Processing CreateTablet for tablet f8993feb727a4e0ebbf083bba3657561 (DEFAULT_TABLE table=test-table [id=eb126534565e453ab51e74dafa595992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:53:59.288632 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8993feb727a4e0ebbf083bba3657561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:59.288740 21673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8993feb727a4e0ebbf083bba3657561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:59.288740 21519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8993feb727a4e0ebbf083bba3657561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:59.295914 21930 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Bootstrap starting.
I20260812 18:53:59.297453 21932 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: Bootstrap starting.
I20260812 18:53:59.298563 21930 tablet_bootstrap.cc:654] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:59.299325 21932 tablet_bootstrap.cc:654] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:59.299772 21930 log.cc:824] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:59.300165 21932 log.cc:824] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:59.300930 21931 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Bootstrap starting.
I20260812 18:53:59.301973 21930 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: No bootstrap required, opened a new log
I20260812 18:53:59.302199 21930 ts_tablet_manager.cc:1397] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:53:59.302836 21931 tablet_bootstrap.cc:654] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:59.303828 21931 log.cc:824] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:59.305341 21930 raft_consensus.cc:348] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.305621 21930 raft_consensus.cc:374] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:59.305697 21930 raft_consensus.cc:729] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6385d2429649428cc2f06e1eaafdc3, State: Initialized, Role: FOLLOWER
I20260812 18:53:59.305764 21932 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: No bootstrap required, opened a new log
I20260812 18:53:59.305994 21932 ts_tablet_manager.cc:1397] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260812 18:53:59.306202 21930 consensus_queue.cc:260] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.306943 21908 heartbeater.cc:503] Master 127.16.40.254:33283 was elected leader, sending a full tablet report...
I20260812 18:53:59.307026 21931 tablet_bootstrap.cc:492] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: No bootstrap required, opened a new log
I20260812 18:53:59.307199 21931 ts_tablet_manager.cc:1397] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:53:59.307279 21930 ts_tablet_manager.cc:1428] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 18:53:59.310855 21931 raft_consensus.cc:348] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.311127 21931 raft_consensus.cc:374] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:59.311196 21931 raft_consensus.cc:729] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a50671d2ef4da193d3a5f91b2868a4, State: Initialized, Role: FOLLOWER
I20260812 18:53:59.311743 21931 consensus_queue.cc:260] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.311745 21932 raft_consensus.cc:348] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.312011 21932 raft_consensus.cc:374] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:59.312080 21932 raft_consensus.cc:729] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67718504b81c47628ea093b47fc8c015, State: Initialized, Role: FOLLOWER
I20260812 18:53:59.312508 21932 consensus_queue.cc:260] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.312964 21748 heartbeater.cc:503] Master 127.16.40.254:33283 was elected leader, sending a full tablet report...
I20260812 18:53:59.313525 21932 ts_tablet_manager.cc:1428] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 18:53:59.312947 21931 ts_tablet_manager.cc:1428] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:53:59.313900 21598 heartbeater.cc:503] Master 127.16.40.254:33283 was elected leader, sending a full tablet report...
I20260812 18:53:59.357014 21938 raft_consensus.cc:482] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:53:59.357360 21938 raft_consensus.cc:504] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.358605 21938 leader_election.cc:288] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67718504b81c47628ea093b47fc8c015 (127.16.40.193:40991), 1f6385d2429649428cc2f06e1eaafdc3 (127.16.40.195:44671)
I20260812 18:53:59.362533 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.358844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:40991 (local address 127.16.40.194:40047)
0812 18:53:59.358999 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.359015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.359124 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.359471 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.359475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.359495 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:59.359619 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.359624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.360376 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.360382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.361248 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.361258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.362079 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.362083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.362208 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.362228 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.362318 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.362392 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 18:53:59.362533 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.359115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:40047 (local address 127.16.40.193:40991)
0812 18:53:59.359247 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.359250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.359262 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:53:59.359311 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.359313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.359351 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.359456 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:59.359750 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.360261 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.361387 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.361954 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.362087 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.362146 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.362335 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.362412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:53:59.362722 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.358987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53909 (local address 127.16.40.195:44671)
0812 18:53:59.359149 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.359154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.359542 (+ 388us) server_negotiation.cc:407] Connection header received
0812 18:53:59.359686 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.359693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.359735 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.359821 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:53:59.360100 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.360620 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.361752 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.362154 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.362286 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.362343 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.362540 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.362619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:53:59.362739 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.358853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:44671 (local address 127.16.40.194:53909)
0812 18:53:59.359441 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.359452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.359574 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.359835 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.359838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.359855 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:53:59.359981 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.359986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.360737 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.360741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.361628 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:53:59.361638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.362286 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.362290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:53:59.362405 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.362428 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.362524 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.362593 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":206,"thread_start_us":164,"threads_started":1}
I20260812 18:53:59.363372 21543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8993feb727a4e0ebbf083bba3657561" candidate_uuid: "21a50671d2ef4da193d3a5f91b2868a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67718504b81c47628ea093b47fc8c015" is_pre_election: true
I20260812 18:53:59.363425 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8993feb727a4e0ebbf083bba3657561" candidate_uuid: "21a50671d2ef4da193d3a5f91b2868a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6385d2429649428cc2f06e1eaafdc3" is_pre_election: true
I20260812 18:53:59.363713 21543 raft_consensus.cc:2454] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21a50671d2ef4da193d3a5f91b2868a4 in term 0.
I20260812 18:53:59.363750 21857 raft_consensus.cc:2454] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21a50671d2ef4da193d3a5f91b2868a4 in term 0.
I20260812 18:53:59.364262 21620 leader_election.cc:302] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6385d2429649428cc2f06e1eaafdc3, 21a50671d2ef4da193d3a5f91b2868a4, 67718504b81c47628ea093b47fc8c015; no voters:
I20260812 18:53:59.364579 21938 raft_consensus.cc:2781] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:53:59.364673 21938 raft_consensus.cc:482] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:53:59.364717 21938 raft_consensus.cc:3037] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:59.365804 21938 raft_consensus.cc:504] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.366164 21938 leader_election.cc:288] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [CANDIDATE]: Term 1 election: Requested vote from peers 67718504b81c47628ea093b47fc8c015 (127.16.40.193:40991), 1f6385d2429649428cc2f06e1eaafdc3 (127.16.40.195:44671)
I20260812 18:53:59.366585 21543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8993feb727a4e0ebbf083bba3657561" candidate_uuid: "21a50671d2ef4da193d3a5f91b2868a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67718504b81c47628ea093b47fc8c015"
I20260812 18:53:59.366598 21857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8993feb727a4e0ebbf083bba3657561" candidate_uuid: "21a50671d2ef4da193d3a5f91b2868a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6385d2429649428cc2f06e1eaafdc3"
I20260812 18:53:59.366699 21543 raft_consensus.cc:3037] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:59.366748 21857 raft_consensus.cc:3037] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:59.367833 21543 raft_consensus.cc:2454] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21a50671d2ef4da193d3a5f91b2868a4 in term 1.
I20260812 18:53:59.367971 21857 raft_consensus.cc:2454] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21a50671d2ef4da193d3a5f91b2868a4 in term 1.
I20260812 18:53:59.368281 21622 leader_election.cc:302] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21a50671d2ef4da193d3a5f91b2868a4, 67718504b81c47628ea093b47fc8c015; no voters:
I20260812 18:53:59.368467 21938 raft_consensus.cc:2781] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:59.368747 21938 raft_consensus.cc:686] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 1 LEADER]: Becoming Leader. State: Replica: 21a50671d2ef4da193d3a5f91b2868a4, State: Running, Role: LEADER
I20260812 18:53:59.369112 21938 consensus_queue.cc:237] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } }
I20260812 18:53:59.372946 21368 catalog_manager.cc:5495] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21a50671d2ef4da193d3a5f91b2868a4 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "21a50671d2ef4da193d3a5f91b2868a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21a50671d2ef4da193d3a5f91b2868a4" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37845 } health_report { overall_health: HEALTHY } } }
I20260812 18:53:59.385566 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.381925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.16.40.194:37845)
0812 18:53:59.382070 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.382073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.382088 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:53:59.382199 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.382202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.382257 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.382353 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:53:59.382584 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.383108 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.383955 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:53:59.384186 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:53:59.384280 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:53:59.384701 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:53:59.385146 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:53:59.385247 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.385371 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.385427 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:53:59.391788 21543 raft_consensus.cc:1261] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 1 FOLLOWER]: Refusing update from remote peer 21a50671d2ef4da193d3a5f91b2868a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:59.392589 21857 raft_consensus.cc:1261] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 1 FOLLOWER]: Refusing update from remote peer 21a50671d2ef4da193d3a5f91b2868a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:53:59.392696 21941 consensus_queue.cc:1035] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67718504b81c47628ea093b47fc8c015" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:59.393194 21938 consensus_queue.cc:1035] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6385d2429649428cc2f06e1eaafdc3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:53:59.402273 21948 mvcc.cc:205] Tried to move back new op lower bound from 7317753198140854272 to 7317753198059364352. Current Snapshot: MvccSnapshot[applied={T|T < 7317753198140854272}]
I20260812 18:53:59.412117 21366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:59.412290 21366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:53:59.414990 21366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6e8f19c86a04770b4bc1cd0c0da08f1: Sending DeleteTablet for 3 replicas of tablet f8993feb727a4e0ebbf083bba3657561
I20260812 18:53:59.415910 21673 tablet_service.cc:1514] Processing DeleteTablet for tablet f8993feb727a4e0ebbf083bba3657561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:59 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:35706
I20260812 18:53:59.415910 21519 tablet_service.cc:1514] Processing DeleteTablet for tablet f8993feb727a4e0ebbf083bba3657561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:59 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:49556
I20260812 18:53:59.415915 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet f8993feb727a4e0ebbf083bba3657561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:53:59 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:58238
I20260812 18:53:59.416487 21957 tablet_replica.cc:330] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: stopping tablet replica
I20260812 18:53:59.416777 21961 tablet_replica.cc:330] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: stopping tablet replica
I20260812 18:53:59.417078 21959 tablet_replica.cc:330] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015: stopping tablet replica
I20260812 18:53:59.417132 21961 raft_consensus.cc:2229] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:53:59.417410 21959 raft_consensus.cc:2229] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:59.417467 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21442
I20260812 18:53:59.417574 21961 raft_consensus.cc:2258] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:59.417773 21959 raft_consensus.cc:2258] T f8993feb727a4e0ebbf083bba3657561 P 67718504b81c47628ea093b47fc8c015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:59.418149 21957 raft_consensus.cc:2229] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:53:59.418430 21957 raft_consensus.cc:2258] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:53:59.420907 21961 ts_tablet_manager.cc:1905] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:59.423401 21957 ts_tablet_manager.cc:1905] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:53:59.423820 21961 ts_tablet_manager.cc:1918] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:53:59.423928 21961 log.cc:1196] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/f8993feb727a4e0ebbf083bba3657561
I20260812 18:53:59.424271 21961 ts_tablet_manager.cc:1939] T f8993feb727a4e0ebbf083bba3657561 P 21a50671d2ef4da193d3a5f91b2868a4: Deleting consensus metadata
I20260812 18:53:59.425577 21352 catalog_manager.cc:4841] TS 21a50671d2ef4da193d3a5f91b2868a4 (127.16.40.194:37845): tablet f8993feb727a4e0ebbf083bba3657561 (table test-table [id=eb126534565e453ab51e74dafa595992]) successfully deleted
I20260812 18:53:59.427258 21957 ts_tablet_manager.cc:1918] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:53:59.427446 21957 log.cc:1196] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/f8993feb727a4e0ebbf083bba3657561
I20260812 18:53:59.427817 21957 ts_tablet_manager.cc:1939] T f8993feb727a4e0ebbf083bba3657561 P 1f6385d2429649428cc2f06e1eaafdc3: Deleting consensus metadata
I20260812 18:53:59.429374 21350 catalog_manager.cc:4841] TS 1f6385d2429649428cc2f06e1eaafdc3 (127.16.40.195:44671): tablet f8993feb727a4e0ebbf083bba3657561 (table test-table [id=eb126534565e453ab51e74dafa595992]) successfully deleted
W20260812 18:53:59.429908 21352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:40991 (error 108)
I20260812 18:53:59.430749 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21603
I20260812 18:53:59.431062 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.430758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:40991 (local address 127.0.0.1:49566)
0812 18:53:59.430902 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.430973 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 18:53:59.435787 21352 catalog_manager.cc:4584] TS 67718504b81c47628ea093b47fc8c015 (127.16.40.193:40991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8993feb727a4e0ebbf083bba3657561: Network error: Client connection negotiation failed: client connection to 127.16.40.193:40991: connect: Connection refused (error 111)
I20260812 18:53:59.441008 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21753
I20260812 18:53:59.454792 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21328
2026-08-12T18:53:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2472 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T18:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:53:59Z Disabled control of system clock
I20260812 18:53:59.492159 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39563
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 18:53:59.611574 21966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:59.611873 21966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:59.611912 21966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:59.616139 21966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:53:59.616214 21966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:59.616231 21966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:59.616248 21966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:53:59.616264 21966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:53:59.621282 21966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:39563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:59.622471 21966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:59.623970 21966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:59.633102 21971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:59.633090 21974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:59.633183 21966 server_base.cc:1034] running on GCE node
W20260812 18:53:59.633090 21972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:59.633920 21966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:59.635078 21966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:59.636255 21966 hybrid_clock.cc:648] HybridClock initialized: now 1786560839636235 us; error 31 us; skew 500 ppm
I20260812 18:53:59.638159 21966 webserver.cc:460] Webserver started at http://127.16.40.254:44609/ using document root <none> and password file <none>
I20260812 18:53:59.638829 21966 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:59.638899 21966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:59.639089 21966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:59.641287 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "ef62e9ab001a4458ac7df4200430ceea"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "af7e12fbef5365e8574452dc044c790b"
server_key_iv: "25db6c7089b27c00a332512ca1e0f547"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.641825 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "ef62e9ab001a4458ac7df4200430ceea"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "af7e12fbef5365e8574452dc044c790b"
server_key_iv: "25db6c7089b27c00a332512ca1e0f547"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.645606 21966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:59.648728 21982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.649991 21966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:53:59.650112 21966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "ef62e9ab001a4458ac7df4200430ceea"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "af7e12fbef5365e8574452dc044c790b"
server_key_iv: "25db6c7089b27c00a332512ca1e0f547"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.650211 21966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:59.655082 21966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:59.655714 21966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:59.655864 21966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:59.663268 21966 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:39563
I20260812 18:53:59.663275 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:39563 every 8 connection(s)
I20260812 18:53:59.664366 21966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:53:59.668154 22040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:53:59.669025 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 21966
I20260812 18:53:59.669142 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:53:59.669476 16547 external_mini_cluster.cc:1428] Setting key 855438d1c5794fc27d6e78f62e665321
I20260812 18:53:59.675568 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: Bootstrap starting.
I20260812 18:53:59.675645 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.671030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.16.40.254:39563)
0812 18:53:59.671724 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.671742 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.671780 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:53:59.672460 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.672479 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.672939 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.673251 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:59.674215 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:59.674221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:59.674234 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:53:59.674247 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:59.674308 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:53:59.674325 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:59.674451 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.674595 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.674885 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":167,"threads_started":1}
I20260812 18:53:59.678265 22040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: Neither blocks nor log segments found. Creating new log.
I20260812 18:53:59.679061 22040 log.cc:824] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: Log is configured to *not* fsync() on all Append() calls
I20260812 18:53:59.681294 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: No bootstrap required, opened a new log
I20260812 18:53:59.685108 22040 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } }
I20260812 18:53:59.685388 22040 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:53:59.685472 22040 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef62e9ab001a4458ac7df4200430ceea, State: Initialized, Role: FOLLOWER
I20260812 18:53:59.686017 22040 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } }
I20260812 18:53:59.686203 22040 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:53:59.686275 22040 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:53:59.686378 22040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:53:59.687417 22040 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } }
I20260812 18:53:59.687799 22040 leader_election.cc:302] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef62e9ab001a4458ac7df4200430ceea; no voters:
I20260812 18:53:59.688148 22040 leader_election.cc:288] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:53:59.688313 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:53:59.688624 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [term 1 LEADER]: Becoming Leader. State: Replica: ef62e9ab001a4458ac7df4200430ceea, State: Running, Role: LEADER
I20260812 18:53:59.688941 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } }
I20260812 18:53:59.689563 22040 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:53:59.690707 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef62e9ab001a4458ac7df4200430ceea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } } }
I20260812 18:53:59.690861 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:59.691304 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:53:59.691257 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef62e9ab001a4458ac7df4200430ceea. Latest consensus state: current_term: 1 leader_uuid: "ef62e9ab001a4458ac7df4200430ceea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef62e9ab001a4458ac7df4200430ceea" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39563 } } }
I20260812 18:53:59.691426 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea [sys.catalog]: This master's current role is: LEADER
I20260812 18:53:59.694487 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:53:59.701308 22055 catalog_manager.cc:1347] Generated new cluster ID: fb981a4268064d3797fd35beaa00e7e2
I20260812 18:53:59.701402 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:53:59.741878 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:53:59.742275 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:53:59.751024 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: Generated new TSK 0
I20260812 18:53:59.752023 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:53:59.767732 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--builtin_ntp_servers=127.16.40.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 18:53:59.891440 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:53:59.891876 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:53:59.891939 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:53:59.895993 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:53:59.896046 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:53:59.896112 22067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:53:59.900882 22067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:53:59.901963 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:53:59.903371 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:53:59.913390 22074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:59.913390 22080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:53:59.913422 22075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:53:59.913681 22067 server_base.cc:1034] running on GCE node
I20260812 18:53:59.914111 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:53:59.914849 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:53:59.916049 22067 hybrid_clock.cc:648] HybridClock initialized: now 1786560839916031 us; error 30 us; skew 500 ppm
I20260812 18:53:59.918177 22067 webserver.cc:460] Webserver started at http://127.16.40.193:39087/ using document root <none> and password file <none>
I20260812 18:53:59.918812 22067 fs_manager.cc:359] Metadata directory not provided
I20260812 18:53:59.918916 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:53:59.919175 22067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:53:59.921413 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "cfa5bb9d7e5a4291885a58ca925e67be"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "09737ae9ae1077f3ea31a5a3fa5b6a73"
server_key_iv: "b9492c2570f6359b2eb78653dbb21c2d"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.922006 22067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "cfa5bb9d7e5a4291885a58ca925e67be"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "09737ae9ae1077f3ea31a5a3fa5b6a73"
server_key_iv: "b9492c2570f6359b2eb78653dbb21c2d"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.925864 22067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:53:59.928483 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.929780 22067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 18:53:59.929924 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "cfa5bb9d7e5a4291885a58ca925e67be"
format_stamp: "Formatted at 2026-08-12 18:53:59 on dist-test-slave-rnrw"
server_key: "09737ae9ae1077f3ea31a5a3fa5b6a73"
server_key_iv: "b9492c2570f6359b2eb78653dbb21c2d"
server_key_version: "encryptionkey@0"
I20260812 18:53:59.930061 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:53:59.934827 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:53:59.935514 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:53:59.935703 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:53:59.936249 22067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:53:59.937254 22067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:53:59.937328 22067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.937397 22067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:53:59.937450 22067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:53:59.946630 22067 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43455
I20260812 18:53:59.946651 22215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43455 every 8 connection(s)
I20260812 18:53:59.947606 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:53:59.953866 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.949604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:38549 (local address 127.16.40.254:39563)
0812 18:53:59.949744 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:53:59.949748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:53:59.950720 (+ 972us) server_negotiation.cc:407] Connection header received
0812 18:53:59.952090 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:53:59.952093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:53:59.952134 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:53:59.952212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:53:59.953350 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:53:59.953352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:53:59.953354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:53:59.953364 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:53:59.953382 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:53:59.953390 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:53:59.953468 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 18:53:59.953655 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 18:53:59.953706 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:53:59.954909 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:53:59.949940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39563 (local address 127.16.40.193:38549)
0812 18:53:59.950589 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 18:53:59.950631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 18:53:59.951878 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:53:59.952376 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:53:59.952382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:53:59.952468 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:53:59.953103 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:53:59.953104 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:53:59.953146 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:53:59.953216 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:53:59.953466 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:53:59.953473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:53:59.953478 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 18:53:59.953700 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 18:53:59.953939 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260812 18:53:59.955619 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22067
I20260812 18:53:59.955752 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:53:59.956048 16547 external_mini_cluster.cc:1428] Setting key 235950c3843a5dd9c01b8f89d0714059
I20260812 18:53:59.956322 22217 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39563
I20260812 18:53:59.956629 22217 heartbeater.cc:464] Registering TS with master...
I20260812 18:53:59.957111 22217 heartbeater.cc:511] Master 127.16.40.254:39563 requested a full tablet report, sending...
I20260812 18:53:59.958191 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--builtin_ntp_servers=127.16.40.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 18:53:59.958840 22002 ts_manager.cc:194] Registered new tserver with Master: cfa5bb9d7e5a4291885a58ca925e67be (127.16.40.193:43455)
W20260812 18:54:00.075822 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:00.076159 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:00.076236 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:00.080371 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:00.080477 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:00.080610 22220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:00.085565 22220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:00.086742 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:00.088233 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:00.098801 22220 server_base.cc:1034] running on GCE node
W20260812 18:54:00.098754 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.098743 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.098717 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:00.099396 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:00.100128 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:00.101320 22220 hybrid_clock.cc:648] HybridClock initialized: now 1786560840101293 us; error 43 us; skew 500 ppm
I20260812 18:54:00.103922 22220 webserver.cc:460] Webserver started at http://127.16.40.194:34339/ using document root <none> and password file <none>
I20260812 18:54:00.104686 22220 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:00.104784 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:00.105064 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:00.108599 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "10424efd8dfa46258a5fc365161f92b9"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "9c01aaec0a8c05cc542d995919f7e48c"
server_key_iv: "9adf46c835f73580acba1f722566bc8a"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.109557 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "10424efd8dfa46258a5fc365161f92b9"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "9c01aaec0a8c05cc542d995919f7e48c"
server_key_iv: "9adf46c835f73580acba1f722566bc8a"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.115283 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 18:54:00.118580 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.120457 22220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:00.120659 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "10424efd8dfa46258a5fc365161f92b9"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "9c01aaec0a8c05cc542d995919f7e48c"
server_key_iv: "9adf46c835f73580acba1f722566bc8a"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.120831 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:00.127123 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:00.127775 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:00.127951 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:00.128613 22220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:00.129624 22220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:00.129681 22220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.129751 22220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:00.129786 22220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.142097 22220 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:36011
I20260812 18:54:00.142158 22366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:36011 every 8 connection(s)
I20260812 18:54:00.143487 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:00.146088 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22220
I20260812 18:54:00.146219 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:00.146575 16547 external_mini_cluster.cc:1428] Setting key b62b80c620a62fe67e07b37333ddcea6
I20260812 18:54:00.148736 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--builtin_ntp_servers=127.16.40.212:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 18:54:00.151923 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.145714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:33229 (local address 127.16.40.254:39563)
0812 18:54:00.145900 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.145906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.147416 (+ 1510us) server_negotiation.cc:407] Connection header received
0812 18:54:00.149588 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.149595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.149648 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.149744 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.151242 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.151249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.151253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.151273 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.151300 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.151314 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.151392 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.151677 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.151732 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:00.153721 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.146265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39563 (local address 127.16.40.194:33229)
0812 18:54:00.147215 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.147283 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.149303 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.149963 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.149974 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.150122 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.150835 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.150839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.150904 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.151018 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.151477 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.151489 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.151496 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.152193 (+ 697us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.152504 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":156,"threads_started":1}
I20260812 18:54:00.155189 22367 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39563
I20260812 18:54:00.155488 22367 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:00.156064 22367 heartbeater.cc:511] Master 127.16.40.254:39563 requested a full tablet report, sending...
I20260812 18:54:00.157572 22002 ts_manager.cc:194] Registered new tserver with Master: 10424efd8dfa46258a5fc365161f92b9 (127.16.40.194:36011)
W20260812 18:54:00.273622 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:00.273953 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:00.274037 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:00.278241 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:00.278334 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:00.278435 22371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:00.283577 22371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:00.284905 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:00.286413 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:00.296716 22378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.296726 22377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.296728 22380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:00.297124 22371 server_base.cc:1034] running on GCE node
I20260812 18:54:00.297549 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:00.298281 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:00.299471 22371 hybrid_clock.cc:648] HybridClock initialized: now 1786560840299455 us; error 27 us; skew 500 ppm
I20260812 18:54:00.301508 22371 webserver.cc:460] Webserver started at http://127.16.40.195:43887/ using document root <none> and password file <none>
I20260812 18:54:00.302155 22371 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:00.302255 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:00.302505 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:00.304785 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "e8b25e830f57432a8e92591ccfb3ea92"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "d38bdc28b9c54046b649f573b83cc4f5"
server_key_iv: "f3287d5eb4c6cca5f1bd3d806989465d"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.305393 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "e8b25e830f57432a8e92591ccfb3ea92"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "d38bdc28b9c54046b649f573b83cc4f5"
server_key_iv: "f3287d5eb4c6cca5f1bd3d806989465d"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.309489 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:00.312314 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.313831 22371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:54:00.313995 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "e8b25e830f57432a8e92591ccfb3ea92"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "d38bdc28b9c54046b649f573b83cc4f5"
server_key_iv: "f3287d5eb4c6cca5f1bd3d806989465d"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.314131 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:00.318991 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:00.319656 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:00.319854 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:00.320433 22371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:00.321444 22371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:00.321516 22371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.321595 22371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:00.321646 22371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.330849 22371 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:42003
I20260812 18:54:00.330914 22513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:42003 every 8 connection(s)
I20260812 18:54:00.331859 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:00.337302 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22371
I20260812 18:54:00.337431 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:00.337786 16547 external_mini_cluster.cc:1428] Setting key f9a1f60293ef6a6c9c63df599216eedf
I20260812 18:54:00.338541 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.333896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:44175 (local address 127.16.40.254:39563)
0812 18:54:00.334034 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.334038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.335168 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 18:54:00.336493 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.336497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.336571 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.336689 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.338035 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.338038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.338040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.338053 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.338074 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.338082 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.338197 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.338351 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.338402 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:54:00.339653 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.334261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39563 (local address 127.16.40.195:44175)
0812 18:54:00.335006 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.335047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.336273 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.336859 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.336866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.336952 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.337702 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.337705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.337762 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.337857 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.338190 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.338198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.338203 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.338391 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.338641 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260812 18:54:00.340873 22514 heartbeater.cc:347] Connected to a master server at 127.16.40.254:39563
I20260812 18:54:00.341140 22514 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:00.341571 22514 heartbeater.cc:511] Master 127.16.40.254:39563 requested a full tablet report, sending...
I20260812 18:54:00.342594 22002 ts_manager.cc:194] Registered new tserver with Master: e8b25e830f57432a8e92591ccfb3ea92 (127.16.40.195:42003)
I20260812 18:54:00.352140 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:54:00.355960 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.354335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.16.40.254:39563)
0812 18:54:00.354470 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.354474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.354657 (+ 183us) server_negotiation.cc:407] Connection header received
0812 18:54:00.354802 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.354804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.354837 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.354903 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.355644 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.355647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.355649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.355660 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.355680 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.355688 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.355759 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.355824 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.355856 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:54:00.359004 22002 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:00.362244 22002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:00.374640 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.371134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:42003 (local address 127.0.0.1:41260)
0812 18:54:00.371629 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.371680 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.371783 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.373144 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.373150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.373228 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.373846 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.373850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.373874 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.373941 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.374277 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.374287 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.374294 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.374413 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.374488 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":81,"threads_started":1}
I20260812 18:54:00.375137 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.371309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.16.40.195:42003)
0812 18:54:00.372660 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.372675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.372705 (+ 30us) server_negotiation.cc:407] Connection header received
0812 18:54:00.372764 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.372773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.372969 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.373124 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.374084 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.374090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.374109 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.374121 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.374158 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.374186 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.374307 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.374894 (+ 587us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.374980 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":139,"threads_started":1}
I20260812 18:54:00.375344 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.371869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:36011 (local address 127.0.0.1:55736)
0812 18:54:00.372644 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.372659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.372751 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.373867 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.373871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.373882 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.374347 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.374350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.374363 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.374393 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.374958 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.374961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.374964 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.375038 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.375075 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":129,"threads_started":1}
I20260812 18:54:00.375869 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.371372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43455 (local address 127.0.0.1:52780)
0812 18:54:00.372250 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.372265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.372352 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.373831 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.373836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.373853 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.375067 (+ 1214us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.375068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.375097 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.375137 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.375626 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.375629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.375632 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.375717 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.375751 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":152,"threads_started":1}
I20260812 18:54:00.375902 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.372206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.16.40.194:36011)
0812 18:54:00.372592 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.372608 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.372756 (+ 148us) server_negotiation.cc:407] Connection header received
0812 18:54:00.372833 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.372837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.372979 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.373125 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.374560 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.374566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.374586 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.374599 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.374642 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.374674 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.375637 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.375690 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.375762 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":130,"threads_started":1}
I20260812 18:54:00.376312 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.372024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.16.40.193:43455)
0812 18:54:00.372668 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.372681 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.372733 (+ 52us) server_negotiation.cc:407] Connection header received
0812 18:54:00.372793 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.372799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.372980 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.373125 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.375278 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.375285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.375295 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.375308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.375433 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.375477 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.375579 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.375767 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.375841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":126,"threads_started":1}
I20260812 18:54:00.378291 22439 tablet_service.cc:1467] Processing CreateTablet for tablet f59b8ed9a4414e82a8af7609a532238b (DEFAULT_TABLE table=test-table [id=bc4b9444960446e59f60636b20c02790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:00.378578 22289 tablet_service.cc:1467] Processing CreateTablet for tablet f59b8ed9a4414e82a8af7609a532238b (DEFAULT_TABLE table=test-table [id=bc4b9444960446e59f60636b20c02790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:00.379477 22439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b8ed9a4414e82a8af7609a532238b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:00.379644 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b8ed9a4414e82a8af7609a532238b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:00.380796 22139 tablet_service.cc:1467] Processing CreateTablet for tablet f59b8ed9a4414e82a8af7609a532238b (DEFAULT_TABLE table=test-table [id=bc4b9444960446e59f60636b20c02790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:00.381848 22139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59b8ed9a4414e82a8af7609a532238b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:00.386507 22535 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Bootstrap starting.
I20260812 18:54:00.387913 22536 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Bootstrap starting.
I20260812 18:54:00.388908 22537 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Bootstrap starting.
I20260812 18:54:00.390213 22536 tablet_bootstrap.cc:654] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:00.390213 22535 tablet_bootstrap.cc:654] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:00.391227 22537 tablet_bootstrap.cc:654] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:00.391206 22536 log.cc:824] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:00.391206 22535 log.cc:824] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:00.392123 22537 log.cc:824] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:00.393442 22535 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: No bootstrap required, opened a new log
I20260812 18:54:00.393445 22536 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: No bootstrap required, opened a new log
I20260812 18:54:00.393651 22536 ts_tablet_manager.cc:1397] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:54:00.393651 22535 ts_tablet_manager.cc:1397] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:54:00.393935 22537 tablet_bootstrap.cc:492] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: No bootstrap required, opened a new log
I20260812 18:54:00.394143 22537 ts_tablet_manager.cc:1397] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 18:54:00.397903 22537 raft_consensus.cc:348] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.398171 22537 raft_consensus.cc:374] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:00.398227 22537 raft_consensus.cc:729] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa5bb9d7e5a4291885a58ca925e67be, State: Initialized, Role: FOLLOWER
I20260812 18:54:00.398366 22536 raft_consensus.cc:348] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.398468 22535 raft_consensus.cc:348] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.398641 22536 raft_consensus.cc:374] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:00.398665 22535 raft_consensus.cc:374] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:00.398705 22536 raft_consensus.cc:729] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10424efd8dfa46258a5fc365161f92b9, State: Initialized, Role: FOLLOWER
I20260812 18:54:00.398711 22535 raft_consensus.cc:729] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b25e830f57432a8e92591ccfb3ea92, State: Initialized, Role: FOLLOWER
I20260812 18:54:00.398727 22537 consensus_queue.cc:260] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.399173 22536 consensus_queue.cc:260] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.399173 22535 consensus_queue.cc:260] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.399554 22217 heartbeater.cc:503] Master 127.16.40.254:39563 was elected leader, sending a full tablet report...
I20260812 18:54:00.400187 22535 ts_tablet_manager.cc:1428] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:54:00.400187 22536 ts_tablet_manager.cc:1428] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:54:00.400527 22367 heartbeater.cc:503] Master 127.16.40.254:39563 was elected leader, sending a full tablet report...
I20260812 18:54:00.400753 22514 heartbeater.cc:503] Master 127.16.40.254:39563 was elected leader, sending a full tablet report...
I20260812 18:54:00.402035 22537 ts_tablet_manager.cc:1428] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260812 18:54:00.448982 22218 tablet.cc:2369] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:54:00.582968 22515 tablet.cc:2369] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:54:00.613060 22542 raft_consensus.cc:482] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:00.613339 22542 raft_consensus.cc:504] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.614616 22542 leader_election.cc:288] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfa5bb9d7e5a4291885a58ca925e67be (127.16.40.193:43455), 10424efd8dfa46258a5fc365161f92b9 (127.16.40.194:36011)
I20260812 18:54:00.616919 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.615035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:42719 (local address 127.16.40.193:43455)
0812 18:54:00.615199 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.615206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.615223 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:54:00.615510 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.615514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.615564 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.615675 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.616451 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.616454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.616457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.616470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.616496 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.616505 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.616678 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.616743 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.616786 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:00.616919 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.614914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43455 (local address 127.16.40.195:42719)
0812 18:54:00.615125 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.615147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.615261 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.615696 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.615699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.615713 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.616287 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.616291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.616305 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.616338 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.616656 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.616660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.616663 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.616745 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.616784 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 18:54:00.617439 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.615275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:36011 (local address 127.16.40.195:43395)
0812 18:54:00.615731 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.615744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:00.615843 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:00.616149 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:00.616155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:00.616173 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:00.616847 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:00.616849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:00.616861 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:00.616891 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:00.617171 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:00.617175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:00.617179 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 18:54:00.617262 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 18:54:00.617298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":179,"threads_started":1}
I20260812 18:54:00.617496 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.615442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43395 (local address 127.16.40.194:36011)
0812 18:54:00.615588 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.615594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.615838 (+ 244us) server_negotiation.cc:407] Connection header received
0812 18:54:00.615956 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.615961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.616005 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.616090 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.617001 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.617007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.617012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.617026 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.617047 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.617058 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.617152 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.617304 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.617354 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:00.617755 22165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b8ed9a4414e82a8af7609a532238b" candidate_uuid: "e8b25e830f57432a8e92591ccfb3ea92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" is_pre_election: true
I20260812 18:54:00.618085 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b8ed9a4414e82a8af7609a532238b" candidate_uuid: "e8b25e830f57432a8e92591ccfb3ea92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10424efd8dfa46258a5fc365161f92b9" is_pre_election: true
I20260812 18:54:00.618172 22165 raft_consensus.cc:2454] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b25e830f57432a8e92591ccfb3ea92 in term 0.
I20260812 18:54:00.618388 22312 raft_consensus.cc:2454] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8b25e830f57432a8e92591ccfb3ea92 in term 0.
I20260812 18:54:00.618821 22388 leader_election.cc:302] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfa5bb9d7e5a4291885a58ca925e67be, e8b25e830f57432a8e92591ccfb3ea92; no voters:
I20260812 18:54:00.619163 22542 raft_consensus.cc:2781] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:00.619247 22542 raft_consensus.cc:482] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:00.619298 22542 raft_consensus.cc:3037] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:00.620512 22542 raft_consensus.cc:504] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.620959 22542 leader_election.cc:288] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [CANDIDATE]: Term 1 election: Requested vote from peers cfa5bb9d7e5a4291885a58ca925e67be (127.16.40.193:43455), 10424efd8dfa46258a5fc365161f92b9 (127.16.40.194:36011)
I20260812 18:54:00.621505 22165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b8ed9a4414e82a8af7609a532238b" candidate_uuid: "e8b25e830f57432a8e92591ccfb3ea92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa5bb9d7e5a4291885a58ca925e67be"
I20260812 18:54:00.621557 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59b8ed9a4414e82a8af7609a532238b" candidate_uuid: "e8b25e830f57432a8e92591ccfb3ea92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10424efd8dfa46258a5fc365161f92b9"
I20260812 18:54:00.621661 22165 raft_consensus.cc:3037] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:00.621675 22312 raft_consensus.cc:3037] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:00.623057 22312 raft_consensus.cc:2454] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b25e830f57432a8e92591ccfb3ea92 in term 1.
I20260812 18:54:00.623059 22165 raft_consensus.cc:2454] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8b25e830f57432a8e92591ccfb3ea92 in term 1.
I20260812 18:54:00.623402 22388 leader_election.cc:302] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10424efd8dfa46258a5fc365161f92b9, e8b25e830f57432a8e92591ccfb3ea92; no voters:
I20260812 18:54:00.623705 22542 raft_consensus.cc:2781] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:00.623952 22542 raft_consensus.cc:686] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 1 LEADER]: Becoming Leader. State: Replica: e8b25e830f57432a8e92591ccfb3ea92, State: Running, Role: LEADER
I20260812 18:54:00.624341 22542 consensus_queue.cc:237] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } }
I20260812 18:54:00.627993 21998 catalog_manager.cc:5495] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8b25e830f57432a8e92591ccfb3ea92 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "e8b25e830f57432a8e92591ccfb3ea92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8b25e830f57432a8e92591ccfb3ea92" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 } health_report { overall_health: UNKNOWN } } }
W20260812 18:54:00.644881 22368 tablet.cc:2369] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:54:00.673440 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.671663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.16.40.195:42003)
0812 18:54:00.671804 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.671808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.671821 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:00.671902 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.671904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.671969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.672048 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.672904 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.672907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.672909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.672920 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.672940 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.672948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.673027 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.673129 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.673165 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:00.679800 22165 raft_consensus.cc:1261] T f59b8ed9a4414e82a8af7609a532238b P cfa5bb9d7e5a4291885a58ca925e67be [term 1 FOLLOWER]: Refusing update from remote peer e8b25e830f57432a8e92591ccfb3ea92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:00.679848 22312 raft_consensus.cc:1261] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 1 FOLLOWER]: Refusing update from remote peer e8b25e830f57432a8e92591ccfb3ea92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:00.680647 22542 consensus_queue.cc:1035] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa5bb9d7e5a4291885a58ca925e67be" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:00.680878 22546 consensus_queue.cc:1035] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10424efd8dfa46258a5fc365161f92b9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:00.690390 22551 mvcc.cc:205] Tried to move back new op lower bound from 7317753203416428544 to 7317753203200344064. Current Snapshot: MvccSnapshot[applied={T|T < 7317753203416428544}]
I20260812 18:54:00.689996 22550 mvcc.cc:205] Tried to move back new op lower bound from 7317753203416428544 to 7317753203200344064. Current Snapshot: MvccSnapshot[applied={T|T < 7317753203416428544}]
I20260812 18:54:00.698468 22001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:00.698701 22001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:00.701777 22001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef62e9ab001a4458ac7df4200430ceea: Sending DeleteTablet for 3 replicas of tablet f59b8ed9a4414e82a8af7609a532238b
I20260812 18:54:00.702904 22289 tablet_service.cc:1514] Processing DeleteTablet for tablet f59b8ed9a4414e82a8af7609a532238b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:00 UTC) from {username='slave'} at 127.0.0.1:55736
I20260812 18:54:00.702929 22139 tablet_service.cc:1514] Processing DeleteTablet for tablet f59b8ed9a4414e82a8af7609a532238b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:00 UTC) from {username='slave'} at 127.0.0.1:52780
I20260812 18:54:00.702963 22439 tablet_service.cc:1514] Processing DeleteTablet for tablet f59b8ed9a4414e82a8af7609a532238b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:00 UTC) from {username='slave'} at 127.0.0.1:41260
I20260812 18:54:00.703608 22560 tablet_replica.cc:330] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: stopping tablet replica
I20260812 18:54:00.703800 22562 tablet_replica.cc:330] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: stopping tablet replica
I20260812 18:54:00.704048 22560 raft_consensus.cc:2229] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:00.704135 22562 raft_consensus.cc:2229] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:00.704324 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22067
I20260812 18:54:00.704591 22562 raft_consensus.cc:2258] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:00.710516 22560 raft_consensus.cc:2258] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:00.712536 22560 ts_tablet_manager.cc:1905] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:00.712982 22562 ts_tablet_manager.cc:1905] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:00.715997 22562 ts_tablet_manager.cc:1918] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:00.716141 22562 log.cc:1196] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/f59b8ed9a4414e82a8af7609a532238b
I20260812 18:54:00.716497 22562 ts_tablet_manager.cc:1939] T f59b8ed9a4414e82a8af7609a532238b P e8b25e830f57432a8e92591ccfb3ea92: Deleting consensus metadata
I20260812 18:54:00.716830 22560 ts_tablet_manager.cc:1918] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:00.717015 22560 log.cc:1196] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/f59b8ed9a4414e82a8af7609a532238b
I20260812 18:54:00.717535 22560 ts_tablet_manager.cc:1939] T f59b8ed9a4414e82a8af7609a532238b P 10424efd8dfa46258a5fc365161f92b9: Deleting consensus metadata
I20260812 18:54:00.718084 21985 catalog_manager.cc:4841] TS e8b25e830f57432a8e92591ccfb3ea92 (127.16.40.195:42003): tablet f59b8ed9a4414e82a8af7609a532238b (table test-table [id=bc4b9444960446e59f60636b20c02790]) successfully deleted
I20260812 18:54:00.719233 21984 catalog_manager.cc:4841] TS 10424efd8dfa46258a5fc365161f92b9 (127.16.40.194:36011): tablet f59b8ed9a4414e82a8af7609a532238b (table test-table [id=bc4b9444960446e59f60636b20c02790]) successfully deleted
W20260812 18:54:00.720723 21984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:43455 (error 108)
I20260812 18:54:00.720853 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22220
I20260812 18:54:00.723471 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.722979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43455 (local address 127.0.0.1:52790)
0812 18:54:00.723272 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:00.723362 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260812 18:54:00.723666 21984 catalog_manager.cc:4584] TS cfa5bb9d7e5a4291885a58ca925e67be (127.16.40.193:43455): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f59b8ed9a4414e82a8af7609a532238b: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43455: connect: Connection refused (error 111)
I20260812 18:54:00.730841 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22371
I20260812 18:54:00.740284 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 21966
2026-08-12T18:54:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1283 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 18:54:00.766805 16547 mini_oidc.cc:152] Starting JWKS server
I20260812 18:54:00.768736 16547 webserver.cc:460] Webserver started at https://127.0.0.1:36877/ using document root <none> and password file <none>
I20260812 18:54:00.768903 16547 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 18:54:00.769142 16547 webserver.cc:460] Webserver started at http://127.0.0.1:43487/ using document root <none> and password file <none>
2026-08-12T18:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:00Z Disabled control of system clock
I20260812 18:54:00.781981 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37751
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:37751
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/testchainca.cert with env {}
W20260812 18:54:00.898577 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:00.898874 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:00.898916 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:00.903020 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:00.903107 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:00.903122 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:00.903141 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:00.903155 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 18:54:00.903172 22569 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 18:54:00.903187 22569 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36877/jwks.json
I20260812 18:54:00.909042 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:37751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:00.910204 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:00.911221 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:00.920068 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.920086 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:00.920095 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:00.920418 22569 server_base.cc:1034] running on GCE node
I20260812 18:54:00.921151 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:00.922307 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:00.923506 22569 hybrid_clock.cc:648] HybridClock initialized: now 1786560840923472 us; error 47 us; skew 500 ppm
I20260812 18:54:00.925673 22569 webserver.cc:460] Webserver started at http://127.16.40.254:38017/ using document root <none> and password file <none>
I20260812 18:54:00.926291 22569 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:00.926390 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:00.926637 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:00.928858 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "d94a870cf3ac4ca89bbedf9f73670498"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "6411d5057113672f68e55c8f4afcc293"
server_key_iv: "c36dbdc7964f4270cde2e0ca1024fd28"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.929437 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "d94a870cf3ac4ca89bbedf9f73670498"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "6411d5057113672f68e55c8f4afcc293"
server_key_iv: "c36dbdc7964f4270cde2e0ca1024fd28"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.933279 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:54:00.935853 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:00.937247 22569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:54:00.937400 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "d94a870cf3ac4ca89bbedf9f73670498"
format_stamp: "Formatted at 2026-08-12 18:54:00 on dist-test-slave-rnrw"
server_key: "6411d5057113672f68e55c8f4afcc293"
server_key_iv: "c36dbdc7964f4270cde2e0ca1024fd28"
server_key_version: "encryptionkey@0"
I20260812 18:54:00.937525 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:00.972802 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:00.973626 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:00.973841 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:00.981285 22569 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:37751
I20260812 18:54:00.981302 22646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:37751 every 8 connection(s)
I20260812 18:54:00.982288 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:00.985795 22647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:00.990331 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22569
I20260812 18:54:00.990469 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:00.990782 16547 external_mini_cluster.cc:1428] Setting key 4e3bff2f5b394d0542cf76a560d6e8b9
I20260812 18:54:00.992172 22647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: Bootstrap starting.
I20260812 18:54:00.994647 22647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:00.995465 22647 log.cc:824] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:00.997449 22647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: No bootstrap required, opened a new log
I20260812 18:54:00.999951 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:00.992224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.16.40.254:37751)
0812 18:54:00.992935 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:00.992950 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:00.992986 (+ 36us) server_negotiation.cc:407] Connection header received
0812 18:54:00.993617 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:00.993633 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:00.994022 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:00.994305 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:00.994780 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:00.995804 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:00.996766 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:00.997236 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:00.997417 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:00.998263 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:00.998269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:00.998282 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:00.998672 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:00.998727 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:00.998747 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:00.998860 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:00.998990 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 18:54:00.999250 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":139,"threads_started":1}
I20260812 18:54:01.001412 22647 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } }
I20260812 18:54:01.001677 22647 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:01.001761 22647 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94a870cf3ac4ca89bbedf9f73670498, State: Initialized, Role: FOLLOWER
I20260812 18:54:01.002310 22647 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } }
I20260812 18:54:01.002476 22647 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:01.002566 22647 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:01.002671 22647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:01.003139 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:54:01.003703 22647 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } }
I20260812 18:54:01.004050 22647 leader_election.cc:302] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94a870cf3ac4ca89bbedf9f73670498; no voters:
I20260812 18:54:01.004418 22647 leader_election.cc:288] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:01.004501 22654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:01.004724 22654 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [term 1 LEADER]: Becoming Leader. State: Replica: d94a870cf3ac4ca89bbedf9f73670498, State: Running, Role: LEADER
I20260812 18:54:01.005036 22654 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } }
I20260812 18:54:01.005816 22647 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:01.006266 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94a870cf3ac4ca89bbedf9f73670498. Latest consensus state: current_term: 1 leader_uuid: "d94a870cf3ac4ca89bbedf9f73670498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } } }
I20260812 18:54:01.006386 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:01.006675 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94a870cf3ac4ca89bbedf9f73670498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94a870cf3ac4ca89bbedf9f73670498" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37751 } } }
I20260812 18:54:01.006772 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498 [sys.catalog]: This master's current role is: LEADER
W20260812 18:54:01.010514 22669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:54:01.010599 22669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:54:01.010689 22658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:01.011412 22658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:01.016741 22658 catalog_manager.cc:1347] Generated new cluster ID: d6dc469d94014a75958224f046b003e2
I20260812 18:54:01.016820 22658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:01.029639 22658 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:01.030656 22658 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:01.039034 22658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94a870cf3ac4ca89bbedf9f73670498: Generated new TSK 0
I20260812 18:54:01.040030 22658 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 18:54:01.078025 22683 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 18:54:01.078241 22683 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 18:54:01.078867 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.059517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.16.40.254:37751)
0812 18:54:01.059674 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:01.059680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:01.059873 (+ 193us) server_negotiation.cc:407] Connection header received
0812 18:54:01.060094 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:01.060098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:01.060163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:01.060290 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0812 18:54:01.060559 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.061421 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.062296 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.062663 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.062774 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.063120 (+ 346us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 18:54:01.078391 (+ 15271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 18:54:01.078710 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 18:54:01.078972 22652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 18:54:01.080533 22682 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:01.059355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37751 (local address 127.0.0.1:43108)
0812 18:54:01.059783 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:01.059796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:01.059913 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:01.060310 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:01.060312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:01.060320 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 18:54:01.060429 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.060434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.061538 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.061541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.062168 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.062177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.062787 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.062790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.062860 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.062909 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 18:54:01.078666 (+ 15757us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 18:54:01.078901 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.254:37751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.40.254:37751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 18:54:01.081960 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22569
2026-08-12T18:54:01Z chronyd exiting
I20260812 18:54:01.123963 16547 test_util.cc:175] -----------------------------------------------
I20260812 18:54:01.124109 16547 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786560811542710-16547-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 18:54:01.125425 16547 mini_oidc.cc:152] Starting JWKS server
I20260812 18:54:01.126349 16547 webserver.cc:460] Webserver started at https://127.0.0.1:42395/ using document root <none> and password file <none>
I20260812 18:54:01.126485 16547 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 18:54:01.126767 16547 webserver.cc:460] Webserver started at http://127.0.0.1:37993/ using document root <none> and password file <none>
2026-08-12T18:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:01Z Disabled control of system clock
I20260812 18:54:01.137650 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39379
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:39379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42395/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/testchainca.cert with env {}
W20260812 18:54:01.254674 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:01.254997 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:01.255043 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:01.259472 22689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:01.259558 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:01.259574 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:01.259593 22689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:01.259608 22689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 18:54:01.259654 22689 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 18:54:01.259690 22689 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42395/jwks.json
I20260812 18:54:01.265421 22689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:39379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42395/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:01.266645 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:01.267889 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:01.277662 22697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:01.277639 22696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:01.277768 22689 server_base.cc:1034] running on GCE node
W20260812 18:54:01.277590 22699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:01.278628 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:01.279942 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:01.281127 22689 hybrid_clock.cc:648] HybridClock initialized: now 1786560841281097 us; error 53 us; skew 500 ppm
I20260812 18:54:01.283216 22689 webserver.cc:460] Webserver started at http://127.16.40.254:41759/ using document root <none> and password file <none>
I20260812 18:54:01.283855 22689 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:01.283957 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:01.284214 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:01.286535 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "7f2c05e54a49455f90dd4af273ee857b"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "ba9c3c5a8c1696a38ee3a516f0055e62"
server_key_iv: "a1a20cfeca87446ecebe00b2a94c6644"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.287150 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "7f2c05e54a49455f90dd4af273ee857b"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "ba9c3c5a8c1696a38ee3a516f0055e62"
server_key_iv: "a1a20cfeca87446ecebe00b2a94c6644"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.291278 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:54:01.294117 22708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:01.295897 22689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:01.296095 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "7f2c05e54a49455f90dd4af273ee857b"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "ba9c3c5a8c1696a38ee3a516f0055e62"
server_key_iv: "a1a20cfeca87446ecebe00b2a94c6644"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.296240 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:01.314168 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:01.314942 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:01.315155 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:01.322490 22689 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:39379
I20260812 18:54:01.322557 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:39379 every 8 connection(s)
I20260812 18:54:01.323539 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:01.324394 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22689
I20260812 18:54:01.324510 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:01.324834 16547 external_mini_cluster.cc:1428] Setting key 90b61670a63cbc89a4c98f3cda2f7448
I20260812 18:54:01.327334 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:01.333186 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b: Bootstrap starting.
I20260812 18:54:01.334746 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.326311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.16.40.254:39379)
0812 18:54:01.327152 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:01.327170 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:01.327208 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:54:01.328033 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:01.328053 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:01.328471 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:01.328908 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:01.329360 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.330373 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.331319 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.331805 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.331980 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.332980 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:01.332985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:01.332999 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:01.333450 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:01.333507 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:01.333529 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:01.333648 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:01.333765 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 18:54:01.334031 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":151,"threads_started":1}
I20260812 18:54:01.335585 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:01.336561 22766 log.cc:824] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:01.337973 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:54:01.338874 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b: No bootstrap required, opened a new log
I20260812 18:54:01.342973 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } }
I20260812 18:54:01.343240 22766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:01.343333 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2c05e54a49455f90dd4af273ee857b, State: Initialized, Role: FOLLOWER
I20260812 18:54:01.343835 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } }
I20260812 18:54:01.343997 22766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:01.344087 22766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:01.344192 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:01.345312 22766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } }
I20260812 18:54:01.345674 22766 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f2c05e54a49455f90dd4af273ee857b; no voters:
I20260812 18:54:01.346074 22766 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:01.346215 22773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:01.346463 22773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [term 1 LEADER]: Becoming Leader. State: Replica: 7f2c05e54a49455f90dd4af273ee857b, State: Running, Role: LEADER
I20260812 18:54:01.346822 22773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } }
I20260812 18:54:01.347474 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:01.348618 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f2c05e54a49455f90dd4af273ee857b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } } }
I20260812 18:54:01.348639 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f2c05e54a49455f90dd4af273ee857b. Latest consensus state: current_term: 1 leader_uuid: "7f2c05e54a49455f90dd4af273ee857b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c05e54a49455f90dd4af273ee857b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 39379 } } }
I20260812 18:54:01.348778 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:01.348778 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:01.349349 22785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:01.352300 22785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:01.358969 22785 catalog_manager.cc:1347] Generated new cluster ID: 814efd47919a44ea82ee1f44730531c3
I20260812 18:54:01.359071 22785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:01.372726 22785 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:01.373822 22785 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:01.383689 22785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f2c05e54a49455f90dd4af273ee857b: Generated new TSK 0
I20260812 18:54:01.384613 22785 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 18:54:01.416785 22802 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 18:54:01.416884 22802 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 18:54:01.417771 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.400711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.16.40.254:39379)
0812 18:54:01.400913 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:01.400918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:01.401090 (+ 172us) server_negotiation.cc:407] Connection header received
0812 18:54:01.401280 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:01.401283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:01.401347 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:01.401481 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0812 18:54:01.401697 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.402529 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.403612 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.403965 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.404076 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.404461 (+ 385us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 18:54:01.417277 (+ 12816us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42395/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 18:54:01.417603 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42395/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":121}
W20260812 18:54:01.417857 22769 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42395/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 18:54:01.417927 22801 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:01.400637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:39379 (local address 127.0.0.1:56086)
0812 18:54:01.400995 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:01.401009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:01.401112 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:01.401478 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:01.401481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:01.401489 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 18:54:01.401590 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.401595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.402689 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.402693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.403458 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.403471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.404137 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.404141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.404216 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.404246 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 18:54:01.417520 (+ 13274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42395/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 18:54:01.417727 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.254:39379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42395/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260812 18:54:01.418653 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22689
2026-08-12T18:54:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 18:54:01.461532 16547 mini_oidc.cc:152] Starting JWKS server
I20260812 18:54:01.462419 16547 webserver.cc:460] Webserver started at https://127.0.0.1:46069/ using document root <none> and password file <none>
I20260812 18:54:01.462525 16547 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 18:54:01.462743 16547 webserver.cc:460] Webserver started at http://127.0.0.1:33113/ using document root <none> and password file <none>
2026-08-12T18:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:01Z Disabled control of system clock
I20260812 18:54:01.474318 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33659
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33659
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 18:54:01.596371 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:01.596807 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:01.596854 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:01.600950 22809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:01.601018 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:01.601035 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:01.601052 22809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:01.601068 22809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 18:54:01.601085 22809 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 18:54:01.601099 22809 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46069/jwks.json
I20260812 18:54:01.605963 22809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:01.607141 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:01.608630 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:01.617834 22817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:01.617853 22809 server_base.cc:1034] running on GCE node
W20260812 18:54:01.617798 22818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:01.617794 22820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:01.618637 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:01.619931 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:01.621150 22809 hybrid_clock.cc:648] HybridClock initialized: now 1786560841621125 us; error 37 us; skew 500 ppm
I20260812 18:54:01.623306 22809 webserver.cc:460] Webserver started at http://127.16.40.254:35129/ using document root <none> and password file <none>
I20260812 18:54:01.623955 22809 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:01.624059 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:01.624316 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:01.626583 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "7bd5a6bd182343348962a49663379a27"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "f6384a442c6942afea6d5e7ca581dd53"
server_key_iv: "a342bf705ea44c7d262937e1770cc868"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.627198 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "7bd5a6bd182343348962a49663379a27"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "f6384a442c6942afea6d5e7ca581dd53"
server_key_iv: "a342bf705ea44c7d262937e1770cc868"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.631215 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:54:01.634140 22827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:01.635452 22809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:01.635608 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "7bd5a6bd182343348962a49663379a27"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "f6384a442c6942afea6d5e7ca581dd53"
server_key_iv: "a342bf705ea44c7d262937e1770cc868"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.635735 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:01.651278 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:01.652061 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:01.652288 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:01.660050 22809 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33659
I20260812 18:54:01.660061 22894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33659 every 8 connection(s)
I20260812 18:54:01.661202 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:01.661525 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22809
I20260812 18:54:01.661659 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:01.661967 16547 external_mini_cluster.cc:1428] Setting key dc12606e06436885c04774568fabf779
I20260812 18:54:01.664793 22895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:01.671689 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27: Bootstrap starting.
I20260812 18:54:01.672122 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.663439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.16.40.254:33659)
0812 18:54:01.664206 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:01.664223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:01.664260 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:54:01.665036 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:01.665055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:01.665459 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:01.665743 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:01.666194 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.667178 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.668417 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.668995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.669147 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.670107 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:01.670113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:01.670126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:01.670533 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:01.670589 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:01.670610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:01.670690 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:01.670960 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 18:54:01.671287 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":159,"threads_started":1}
I20260812 18:54:01.674105 22895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:01.674922 22895 log.cc:824] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:01.675235 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:54:01.677289 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27: No bootstrap required, opened a new log
I20260812 18:54:01.680868 22895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } }
I20260812 18:54:01.681157 22895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:01.681197 22895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd5a6bd182343348962a49663379a27, State: Initialized, Role: FOLLOWER
I20260812 18:54:01.681703 22895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } }
I20260812 18:54:01.681872 22895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:01.682024 22895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:01.682157 22895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:01.683185 22895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } }
I20260812 18:54:01.683555 22895 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd5a6bd182343348962a49663379a27; no voters:
I20260812 18:54:01.683902 22895 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:01.684074 22900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:01.684350 22900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd5a6bd182343348962a49663379a27, State: Running, Role: LEADER
I20260812 18:54:01.684695 22900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } }
I20260812 18:54:01.685323 22895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:01.686678 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd5a6bd182343348962a49663379a27. Latest consensus state: current_term: 1 leader_uuid: "7bd5a6bd182343348962a49663379a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } } }
I20260812 18:54:01.686861 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:01.687038 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd5a6bd182343348962a49663379a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd5a6bd182343348962a49663379a27" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33659 } } }
I20260812 18:54:01.687148 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:01.687620 22917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:01.690821 22917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:01.696442 22917 catalog_manager.cc:1347] Generated new cluster ID: 8e175dd6dd4c4497a8972a85c7adf67a
I20260812 18:54:01.696528 22917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:01.710059 22917 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:01.711606 22917 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:01.727257 22917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd5a6bd182343348962a49663379a27: Generated new TSK 0
I20260812 18:54:01.728303 22917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:54:01.742395 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.736269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.16.40.254:33659)
0812 18:54:01.736416 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:01.736421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:01.736574 (+ 153us) server_negotiation.cc:407] Connection header received
0812 18:54:01.736791 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:01.736795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:01.736863 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:01.737017 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0812 18:54:01.737258 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.738215 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.739130 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.739540 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.739658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.740033 (+ 375us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 18:54:01.741789 (+ 1756us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 18:54:01.742237 (+ 448us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47004: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 18:54:01.742477 22898 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47004: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 18:54:01.744220 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:01.736155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33659 (local address 127.0.0.1:47004)
0812 18:54:01.736442 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:01.736464 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:01.736622 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:01.737032 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:01.737034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:01.737047 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0812 18:54:01.737142 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.737146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.738354 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.738356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.739012 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:01.739021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:01.739686 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:01.739690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:01.739778 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:01.739814 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 18:54:01.742003 (+ 2189us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 18:54:01.744010 (+ 2007us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.254:33659: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1767,"thread_start_us":111,"threads_started":1}
I20260812 18:54:01.744979 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22809
E20260812 18:54:01.754691 22935 webserver.cc:533] Webserver: sslize failed:
E20260812 18:54:01.754809 22935 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T18:54:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T18:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:01Z Disabled control of system clock
I20260812 18:54:01.799782 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33417
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33417
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/testchainca.cert with env {}
W20260812 18:54:01.920143 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:01.920445 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:01.920490 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:01.924702 22939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:01.924780 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:01.924796 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:01.924813 22939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:01.924829 22939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 18:54:01.924845 22939 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 18:54:01.924908 22939 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 18:54:01.930663 22939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:01.931865 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:01.932981 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:01.942044 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:01.942063 22944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:01.942080 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:01.942675 22939 server_base.cc:1034] running on GCE node
I20260812 18:54:01.943324 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:01.944622 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:01.945812 22939 hybrid_clock.cc:648] HybridClock initialized: now 1786560841945774 us; error 53 us; skew 500 ppm
I20260812 18:54:01.948007 22939 webserver.cc:460] Webserver started at http://127.16.40.254:34265/ using document root <none> and password file <none>
I20260812 18:54:01.948716 22939 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:01.948793 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:01.949070 22939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:01.951206 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "ad645c51b9e34898ab364cb19c6f06ed"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "6e8d7fa65b81256af4f031265bc579c7"
server_key_iv: "45fbb691fa1ee50f77c1a077b7cfc0d3"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.951850 22939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "ad645c51b9e34898ab364cb19c6f06ed"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "6e8d7fa65b81256af4f031265bc579c7"
server_key_iv: "45fbb691fa1ee50f77c1a077b7cfc0d3"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.955904 22939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:54:01.958631 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:01.959977 22939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 18:54:01.960136 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "ad645c51b9e34898ab364cb19c6f06ed"
format_stamp: "Formatted at 2026-08-12 18:54:01 on dist-test-slave-rnrw"
server_key: "6e8d7fa65b81256af4f031265bc579c7"
server_key_iv: "45fbb691fa1ee50f77c1a077b7cfc0d3"
server_key_version: "encryptionkey@0"
I20260812 18:54:01.960269 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:01.982846 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:01.983705 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:01.983946 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:01.991752 22939 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33417
I20260812 18:54:01.991771 23019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33417 every 8 connection(s)
I20260812 18:54:01.992844 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:01.996013 23020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:01.997412 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 22939
I20260812 18:54:01.997576 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:01.997895 16547 external_mini_cluster.cc:1428] Setting key 44a7558c71ab0f40deda1b0c71ef53ed
I20260812 18:54:02.002597 23020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed: Bootstrap starting.
I20260812 18:54:02.005117 23020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:02.006064 23020 log.cc:824] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:02.008188 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:01.999460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.16.40.254:33417)
0812 18:54:02.000193 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:02.000210 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:02.000249 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:54:02.001016 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:02.001035 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:02.001458 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:02.001815 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:02.002274 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.003337 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.004289 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.004865 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.005056 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:02.005966 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:02.005971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:02.005986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:02.006454 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:02.006559 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:02.006591 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:02.006705 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:02.006881 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 18:54:02.007189 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
I20260812 18:54:02.008620 23020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed: No bootstrap required, opened a new log
I20260812 18:54:02.011551 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:54:02.012887 23020 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } }
I20260812 18:54:02.013130 23020 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:02.013198 23020 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad645c51b9e34898ab364cb19c6f06ed, State: Initialized, Role: FOLLOWER
I20260812 18:54:02.013644 23020 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } }
I20260812 18:54:02.013767 23020 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:02.013856 23020 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:02.013978 23020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:02.014988 23020 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } }
I20260812 18:54:02.015357 23020 leader_election.cc:302] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad645c51b9e34898ab364cb19c6f06ed; no voters:
I20260812 18:54:02.015725 23020 leader_election.cc:288] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:02.015985 23026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:02.016199 23026 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [term 1 LEADER]: Becoming Leader. State: Replica: ad645c51b9e34898ab364cb19c6f06ed, State: Running, Role: LEADER
I20260812 18:54:02.016505 23026 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } }
I20260812 18:54:02.017138 23020 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:02.018458 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad645c51b9e34898ab364cb19c6f06ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } } }
I20260812 18:54:02.018626 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:02.018965 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad645c51b9e34898ab364cb19c6f06ed. Latest consensus state: current_term: 1 leader_uuid: "ad645c51b9e34898ab364cb19c6f06ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad645c51b9e34898ab364cb19c6f06ed" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33417 } } }
I20260812 18:54:02.019071 23039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:02.019089 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:02.022485 23039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:02.029083 23039 catalog_manager.cc:1347] Generated new cluster ID: 67749dfd800c449fb7feff573d64ec76
I20260812 18:54:02.029189 23039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:02.047922 23039 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:02.049019 23039 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:02.057983 23039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad645c51b9e34898ab364cb19c6f06ed: Generated new TSK 0
I20260812 18:54:02.058813 23039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:54:02.527776 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:02.078156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.16.40.254:33417)
0812 18:54:02.078318 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:02.078323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:02.078480 (+ 157us) server_negotiation.cc:407] Connection header received
0812 18:54:02.078673 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:02.078676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:02.078732 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:02.078833 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0812 18:54:02.079171 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.079777 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.081097 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.081343 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.081434 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:02.081801 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 18:54:02.527173 (+445372us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 18:54:02.527600 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 18:54:02.527889 23059 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:02.077952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33417 (local address 127.0.0.1:51582)
0812 18:54:02.078365 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:02.078381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:02.078525 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:02.078861 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:02.078865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:02.078874 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 18:54:02.079003 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:02.079010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.079902 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.079906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:02.080957 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:02.080969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:02.081481 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:02.081485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:02.081582 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:02.081614 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 18:54:02.527475 (+445861us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 18:54:02.527687 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.40.254:33417: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
W20260812 18:54:02.527899 23024 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51582: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260812 18:54:02.528782 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 22939
2026-08-12T18:54:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (765 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (141 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (140 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:02 dist-test-slave-rnrw krb5kdc[23073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:02 dist-test-slave-rnrw krb5kdc[23073](info): set up 2 sockets
Aug 12 18:54:02 dist-test-slave-rnrw krb5kdc[23073](info): commencing operation
krb5kdc: starting...
W20260812 18:54:03.954375 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:54:04.095361 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:43213
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:04.215318 23089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:04.215689 23089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:04.215781 23089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:04.220022 23089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:04.220157 23089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:04.220197 23089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:04.220239 23089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:04.220276 23089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:04.226104 23089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:43213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:04.227423 23089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:04.228600 23089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:04.237982 23089 server_base.cc:1034] running on GCE node
W20260812 18:54:04.237936 23097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:04.237949 23096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:04.237941 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:04.238866 23089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:04.240167 23089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:04.241375 23089 hybrid_clock.cc:648] HybridClock initialized: now 1786560844241323 us; error 68 us; skew 500 ppm
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:04.244393 23089 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:54:04.245384 23089 webserver.cc:460] Webserver started at http://127.16.40.254:38291/ using document root <none> and password file <none>
I20260812 18:54:04.245981 23089 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:04.246111 23089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:04.246356 23089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:04.248523 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "e04d4e998bb8427b946270621e7e21ab"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "e15733ca6c267d09a32384f271609f34"
server_key_iv: "a17bf64f0ce3e30f9d5ab9e230cf867f"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.249217 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "e04d4e998bb8427b946270621e7e21ab"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "e15733ca6c267d09a32384f271609f34"
server_key_iv: "a17bf64f0ce3e30f9d5ab9e230cf867f"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.253501 23089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:54:04.256242 23108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.257750 23089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:54:04.257915 23089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "e04d4e998bb8427b946270621e7e21ab"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "e15733ca6c267d09a32384f271609f34"
server_key_iv: "a17bf64f0ce3e30f9d5ab9e230cf867f"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.258059 23089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:04.287290 23089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:04.290689 23089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:04.290921 23089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:04.298524 23089 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:43213
I20260812 18:54:04.298559 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:43213 every 8 connection(s)
I20260812 18:54:04.299548 23089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:04.302561 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:04.303936 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23089
I20260812 18:54:04.304080 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:04.304395 16547 external_mini_cluster.cc:1428] Setting key cb7d19e0460c57238909aed85b4ab51e
I20260812 18:54:04.309088 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab: Bootstrap starting.
I20260812 18:54:04.311434 23177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:04.312232 23177 log.cc:824] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:04.314647 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab: No bootstrap required, opened a new log
I20260812 18:54:04.318341 23177 raft_consensus.cc:348] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } }
I20260812 18:54:04.318634 23177 raft_consensus.cc:374] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:04.318717 23177 raft_consensus.cc:729] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04d4e998bb8427b946270621e7e21ab, State: Initialized, Role: FOLLOWER
I20260812 18:54:04.319232 23177 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } }
I20260812 18:54:04.319458 23177 raft_consensus.cc:388] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:04.319551 23177 raft_consensus.cc:482] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:04.319684 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:04.320026 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:04.306081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.16.40.254:43213)
0812 18:54:04.306623 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:04.306639 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:04.306677 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:54:04.307322 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:04.307338 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:04.307695 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:04.308089 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:04.309000 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.310075 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.310961 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.311459 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.311676 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:04.314646 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:04.314653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:04.314669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:04.314720 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:04.316680 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.317211 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.317217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.317222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.317361 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.317686 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.317690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.317693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.318165 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:04.318411 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:04.318674 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 18:54:04.319033 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":147,"threads_started":1}
I20260812 18:54:04.320924 23177 raft_consensus.cc:504] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } }
I20260812 18:54:04.321291 23177 leader_election.cc:302] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04d4e998bb8427b946270621e7e21ab; no voters:
I20260812 18:54:04.321677 23177 leader_election.cc:288] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:04.321774 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:04.321956 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [term 1 LEADER]: Becoming Leader. State: Replica: e04d4e998bb8427b946270621e7e21ab, State: Running, Role: LEADER
I20260812 18:54:04.322333 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } }
I20260812 18:54:04.323091 23177 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:04.324678 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04d4e998bb8427b946270621e7e21ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } } }
I20260812 18:54:04.325026 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:04.324928 23186 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04d4e998bb8427b946270621e7e21ab. Latest consensus state: current_term: 1 leader_uuid: "e04d4e998bb8427b946270621e7e21ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04d4e998bb8427b946270621e7e21ab" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 43213 } } }
I20260812 18:54:04.325380 23186 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:04.325695 23200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:04.329018 23200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:04.335490 23200 catalog_manager.cc:1347] Generated new cluster ID: 015f25ee56f14547a799c20540ef768d
I20260812 18:54:04.335625 23200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:04.368302 23200 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:04.369370 23200 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:04.377087 23200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e04d4e998bb8427b946270621e7e21ab: Generated new TSK 0
I20260812 18:54:04.377866 23200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:54:04.416792 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:04.549154 23209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:04.549477 23209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:04.549527 23209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:04.553748 23209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:04.553817 23209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:04.553879 23209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:04.559206 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:04.560302 23209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:04.561357 23209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:04.571647 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:04.571826 23209 server_base.cc:1034] running on GCE node
W20260812 18:54:04.571676 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:04.571702 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:04.572626 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:04.573412 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:04.574770 23209 hybrid_clock.cc:648] HybridClock initialized: now 1786560844574746 us; error 34 us; skew 500 ppm
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:04.578192 23209 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:54:04.579276 23209 webserver.cc:460] Webserver started at http://127.16.40.193:33657/ using document root <none> and password file <none>
I20260812 18:54:04.579988 23209 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:04.580099 23209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:04.580371 23209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:04.582774 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "599243133b0044d8a32c47a74984d5f6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "968292f78e4b8a48784e74e261f3a883"
server_key_iv: "6646174ea7b7d72cb8b82cb07d4d1ddc"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.583431 23209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "599243133b0044d8a32c47a74984d5f6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "968292f78e4b8a48784e74e261f3a883"
server_key_iv: "6646174ea7b7d72cb8b82cb07d4d1ddc"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.587836 23209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260812 18:54:04.590826 23228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.592140 23209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:04.592304 23209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "599243133b0044d8a32c47a74984d5f6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "968292f78e4b8a48784e74e261f3a883"
server_key_iv: "6646174ea7b7d72cb8b82cb07d4d1ddc"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.592442 23209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:04.619192 23209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:04.622749 23209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:04.622988 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:04.623719 23209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:04.624872 23209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:04.624953 23209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.625023 23209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:04.625061 23209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.635056 23209 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:41403
I20260812 18:54:04.635083 23368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:41403 every 8 connection(s)
I20260812 18:54:04.636219 23209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:04.645526 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23209
I20260812 18:54:04.645697 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:04.646109 16547 external_mini_cluster.cc:1428] Setting key bca8b8dda461a06252645ec84bd982a9
I20260812 18:54:04.651254 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:04.638209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:60199 (local address 127.16.40.254:43213)
0812 18:54:04.638353 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:04.638356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:04.639204 (+ 848us) server_negotiation.cc:407] Connection header received
0812 18:54:04.640664 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:04.640668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:04.640731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:04.640835 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:04.642035 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.642569 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.643724 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.643968 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.644086 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:04.646785 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:04.646790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:04.646793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:04.646824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:04.648656 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.649322 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.649324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.649326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.649412 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.649882 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.649887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.649890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.650190 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:04.650315 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:04.650848 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 18:54:04.651040 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:54:04.652105 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:04.638556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:43213 (local address 127.16.40.193:60199)
0812 18:54:04.639030 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:04.639084 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:04.640397 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:04.641015 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:04.641022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:04.641395 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:04.641871 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:04.641885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.642709 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.642712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:04.643587 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:04.643597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.644142 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.644146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:04.644327 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:04.644977 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:04.644994 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:04.646562 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:04.648783 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:04.648792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:04.648812 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:04.649206 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:04.649505 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:04.649510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:04.649515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:04.649731 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:04.650319 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:04.650325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:04.650591 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 18:54:04.650758 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 18:54:04.651034 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260812 18:54:04.653600 23369 heartbeater.cc:347] Connected to a master server at 127.16.40.254:43213
I20260812 18:54:04.653885 23369 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:04.654457 23369 heartbeater.cc:511] Master 127.16.40.254:43213 requested a full tablet report, sending...
I20260812 18:54:04.656081 23123 ts_manager.cc:194] Registered new tserver with Master: 599243133b0044d8a32c47a74984d5f6 (127.16.40.193:41403)
I20260812 18:54:04.657315 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:60199
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:54:04.679665 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:04.802799 23375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:04.803148 23375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:04.803226 23375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:04.807483 23375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:04.807590 23375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:04.807690 23375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:04.813268 23375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:04.814467 23375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:04.815601 23375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:04.826413 23381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:04.826458 23375 server_base.cc:1034] running on GCE node
W20260812 18:54:04.826416 23380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:04.826413 23383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:04.827123 23375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:04.827890 23375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:04.829066 23375 hybrid_clock.cc:648] HybridClock initialized: now 1786560844829044 us; error 34 us; skew 500 ppm
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:04.832109 23375 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:54:04.833206 23375 webserver.cc:460] Webserver started at http://127.16.40.194:41387/ using document root <none> and password file <none>
I20260812 18:54:04.833835 23375 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:04.833936 23375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:04.834197 23375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:04.836463 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "06a3faa79126484bb2d77757eea520e6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "2019075f1240824cea99090e5cd62c83"
server_key_iv: "253ffdc6eaaf8af87b55cef9abacdb0a"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.837168 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "06a3faa79126484bb2d77757eea520e6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "2019075f1240824cea99090e5cd62c83"
server_key_iv: "253ffdc6eaaf8af87b55cef9abacdb0a"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.841440 23375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:04.844321 23395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.846235 23375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:04.846385 23375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "06a3faa79126484bb2d77757eea520e6"
format_stamp: "Formatted at 2026-08-12 18:54:04 on dist-test-slave-rnrw"
server_key: "2019075f1240824cea99090e5cd62c83"
server_key_iv: "253ffdc6eaaf8af87b55cef9abacdb0a"
server_key_version: "encryptionkey@0"
I20260812 18:54:04.846521 23375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:04.866947 23375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:04.870472 23375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:04.870764 23375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:04.871426 23375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:04.872375 23375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:04.872462 23375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:04.872536 23375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:04.872603 23375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 18:54:04.882069 23375 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33419
I20260812 18:54:04.882089 23533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33419 every 8 connection(s)
I20260812 18:54:04.883026 23375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:04.888001 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23375
I20260812 18:54:04.888149 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:04.888432 16547 external_mini_cluster.cc:1428] Setting key 0a332d75386aa866c0b3232476fc06a9
Aug 12 18:54:04 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560844, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:04.900087 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:04.885182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:44213 (local address 127.16.40.254:43213)
0812 18:54:04.885339 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:04.885344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:04.886239 (+ 895us) server_negotiation.cc:407] Connection header received
0812 18:54:04.887834 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:04.887838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:04.887906 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:04.888028 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:04.889590 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.890212 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.892012 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.892240 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.892369 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:04.895265 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:04.895272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:04.895278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:04.895315 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:04.897360 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.898135 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.898140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.898141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.898245 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:04.898658 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:04.898665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:04.898668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:04.898946 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:04.899068 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:04.899736 (+ 668us) server_negotiation.cc:299] Negotiation successful
0812 18:54:04.899888 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:54:04.901027 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:04.885559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:43213 (local address 127.16.40.194:44213)
0812 18:54:04.886062 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:04.886110 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:04.887541 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:04.888240 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:04.888252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:04.888792 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:04.889414 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:04.889436 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.890342 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.890346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:04.891866 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:04.891881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:04.892417 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:04.892422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:04.892674 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:04.893319 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:04.893336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:04.895052 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:04.897583 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:04.897616 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:04.897637 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:04.898003 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:04.898358 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:04.898360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:04.898362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:04.898545 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:04.899089 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:04.899095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:04.899421 (+ 326us) client_negotiation.cc:769] Sending connection context
0812 18:54:04.899632 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 18:54:04.899943 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260812 18:54:04.902685 23534 heartbeater.cc:347] Connected to a master server at 127.16.40.254:43213
I20260812 18:54:04.902984 23534 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:54:04.903654 23534 heartbeater.cc:511] Master 127.16.40.254:43213 requested a full tablet report, sending...
I20260812 18:54:04.905124 23123 ts_manager.cc:194] Registered new tserver with Master: 06a3faa79126484bb2d77757eea520e6 (127.16.40.194:33419)
I20260812 18:54:04.906177 23123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:44213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:54:04.925239 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:05.047314 23540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:05.047636 23540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:05.047719 23540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:05.052050 23540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:05.052131 23540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:05.052222 23540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:05.057891 23540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:05.059139 23540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:05.060242 23540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:05.070730 23550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:05.070742 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:05.070755 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:05.071069 23540 server_base.cc:1034] running on GCE node
I20260812 18:54:05.071693 23540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:05.072429 23540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:05.073695 23540 hybrid_clock.cc:648] HybridClock initialized: now 1786560845073680 us; error 28 us; skew 500 ppm
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.076901 23540 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:54:05.077914 23540 webserver.cc:460] Webserver started at http://127.16.40.195:37231/ using document root <none> and password file <none>
I20260812 18:54:05.078544 23540 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:05.078645 23540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:05.078891 23540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:05.081164 23540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "f4e0917568e24d01be4ff7c60b3f1b92"
format_stamp: "Formatted at 2026-08-12 18:54:05 on dist-test-slave-rnrw"
server_key: "c22812a3bac35f815b571d1dcfa7e4f3"
server_key_iv: "8452312cbcf13de04e6ccb212f314c99"
server_key_version: "encryptionkey@0"
I20260812 18:54:05.081780 23540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "f4e0917568e24d01be4ff7c60b3f1b92"
format_stamp: "Formatted at 2026-08-12 18:54:05 on dist-test-slave-rnrw"
server_key: "c22812a3bac35f815b571d1dcfa7e4f3"
server_key_iv: "8452312cbcf13de04e6ccb212f314c99"
server_key_version: "encryptionkey@0"
I20260812 18:54:05.085999 23540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:05.088794 23561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:05.090142 23540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:05.090298 23540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "f4e0917568e24d01be4ff7c60b3f1b92"
format_stamp: "Formatted at 2026-08-12 18:54:05 on dist-test-slave-rnrw"
server_key: "c22812a3bac35f815b571d1dcfa7e4f3"
server_key_iv: "8452312cbcf13de04e6ccb212f314c99"
server_key_version: "encryptionkey@0"
I20260812 18:54:05.090430 23540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:05.116422 23540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:05.120079 23540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:05.120306 23540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:05.121052 23540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:05.121939 23540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:05.122017 23540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:05.122092 23540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:05.122145 23540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:05.131672 23540 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:37487
I20260812 18:54:05.131701 23703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:37487 every 8 connection(s)
I20260812 18:54:05.132661 23540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:05.133811 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23540
I20260812 18:54:05.134006 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:05.134306 16547 external_mini_cluster.cc:1428] Setting key e802388990e975ab717d3737e58dced9
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:05.146692 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:05.134743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:44543 (local address 127.16.40.254:43213)
0812 18:54:05.134889 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.134894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.135863 (+ 969us) server_negotiation.cc:407] Connection header received
0812 18:54:05.137192 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.137195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.137243 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.137334 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.138563 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.139086 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.140191 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.140394 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.140472 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:05.143103 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:05.143107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:05.143109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:05.143134 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:05.144734 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.145241 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.145244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.145245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.145323 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.145631 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.145633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.145634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.145857 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:05.145937 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:05.146429 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 18:54:05.146563 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:54:05.147624 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:05.135197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:43213 (local address 127.16.40.195:44543)
0812 18:54:05.135706 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:05.135760 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:05.136997 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:05.137477 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:05.137483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:05.137890 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:05.138415 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:05.138428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.139211 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.139214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:05.140082 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:05.140092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.140523 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.140526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:05.140740 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:05.141285 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:05.141301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:05.142928 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:05.144851 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:05.144857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:05.144870 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:05.145153 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:05.145404 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:05.145406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:05.145408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:05.145547 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:05.145950 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:05.145955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:05.146220 (+ 265us) client_negotiation.cc:769] Sending connection context
0812 18:54:05.146382 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 18:54:05.146716 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260812 18:54:05.148994 23704 heartbeater.cc:347] Connected to a master server at 127.16.40.254:43213
I20260812 18:54:05.149302 23704 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:05.149873 23704 heartbeater.cc:511] Master 127.16.40.254:43213 requested a full tablet report, sending...
I20260812 18:54:05.150966 23127 ts_manager.cc:194] Registered new tserver with Master: f4e0917568e24d01be4ff7c60b3f1b92 (127.16.40.195:37487)
I20260812 18:54:05.151670 23127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:44543
I20260812 18:54:05.160712 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/test-admin.keytab.
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.178418 16547 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 18:54:05.181710 23713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:05.180002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.16.40.193:41403)
0812 18:54:05.180308 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.180314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.180339 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:54:05.180391 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.180397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.180700 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.180831 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.181505 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56576: BlockingRecv error: recv got EOF from 127.0.0.1:56576 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260812 18:54:05.181759 16547 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc' of type FILE
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.182899 16547 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.183923 16547 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:54:05.194792 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:05.185120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.16.40.193:41403)
0812 18:54:05.185265 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.185269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.185302 (+ 33us) server_negotiation.cc:407] Connection header received
0812 18:54:05.185462 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.185465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.185513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.185591 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.186189 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.186966 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.187890 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.188340 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.188440 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:05.191098 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:05.191110 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:05.191127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:05.191154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:05.192822 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.193285 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.193290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.193292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.193375 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.193663 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.193670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.193671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.193938 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:05.194115 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:05.194425 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 18:54:05.194598 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 18:54:05.198977 23713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:05.197924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.16.40.193:41403)
0812 18:54:05.198071 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.198075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.198204 (+ 129us) server_negotiation.cc:407] Connection header received
0812 18:54:05.198285 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.198287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.198323 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.198398 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.198879 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56596: BlockingRecv error: recv got EOF from 127.0.0.1:56596 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 18:54:05.199111 16547 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc' of type FILE
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.200202 16547 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.201166 16547 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:54:05.210247 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:05.202017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.16.40.193:41403)
0812 18:54:05.202169 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.202172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.202218 (+ 46us) server_negotiation.cc:407] Connection header received
0812 18:54:05.202321 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.202323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.202359 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.202426 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.202908 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.203417 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.204196 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.204415 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.204601 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:05.206826 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:05.206829 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:05.206831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:05.206856 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:05.208505 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.208988 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.208991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.208992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.209080 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.209346 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.209350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.209351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.209583 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:05.209715 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:05.209893 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 18:54:05.210006 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 18:54:05.213335 23713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:05.212194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.16.40.193:41403)
0812 18:54:05.212339 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.212343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.212458 (+ 115us) server_negotiation.cc:407] Connection header received
0812 18:54:05.212664 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.212667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.212717 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.212798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.213224 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56614: BlockingRecv error: recv got EOF from 127.0.0.1:56614 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 18:54:05.213470 16547 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786560811542710-16547-0/krb5kdc/krb5cc' of type FILE
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.214505 16547 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:05.215384 16547 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:54:05.224043 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:05.216250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.16.40.193:41403)
0812 18:54:05.216376 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:05.216380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:05.216393 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:05.216504 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:05.216507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:05.216608 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:05.216674 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:05.217113 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.217599 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.218370 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:05.218597 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:05.218699 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:05.221015 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:05.221019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:05.221020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:05.221042 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:05.222445 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.222845 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.222848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.222850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.222927 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:05.223179 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:05.223181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:05.223182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:05.223400 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:05.223522 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:05.223716 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 18:54:05.223836 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:54:05.225234 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23209
I20260812 18:54:05.235342 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23375
I20260812 18:54:05.244695 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23540
I20260812 18:54:05.262791 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23089
2026-08-12T18:54:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2451 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23727](info): set up 2 sockets
Aug 12 18:54:05 dist-test-slave-rnrw krb5kdc[23727](info): commencing operation
krb5kdc: starting...
W20260812 18:54:06.402397 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:06 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab.
Entry for principal oryx/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab.
I20260812 18:54:06.544968 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41697
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab
--principal=oryx/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:06.662112 23741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:06.662426 23741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:06.662479 23741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:06.666739 23741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:06.666821 23741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:06.666841 23741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:06.666857 23741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:06.666873 23741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:06.672525 23741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab
--principal=oryx/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:06.673885 23741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:06.675091 23741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:06.684399 23749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:06.684438 23746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:06.684592 23741 server_base.cc:1034] running on GCE node
W20260812 18:54:06.684408 23747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:06.685485 23741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:06.686849 23741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:06.688027 23741 hybrid_clock.cc:648] HybridClock initialized: now 1786560846688007 us; error 36 us; skew 500 ppm
Aug 12 18:54:06 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:06.691128 23741 init.cc:377] Logged in from keytab as oryx/127.16.40.254@KRBTEST.COM (short username oryx)
I20260812 18:54:06.692114 23741 webserver.cc:460] Webserver started at http://127.16.40.254:44793/ using document root <none> and password file <none>
I20260812 18:54:06.692764 23741 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:06.692834 23741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:06.693020 23741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:06.695281 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "869ed47ea6194f9194abb85fcd53ea45"
format_stamp: "Formatted at 2026-08-12 18:54:06 on dist-test-slave-rnrw"
server_key: "4346c9be1de7ae959e9a64b8d1122abc"
server_key_iv: "cf0ab733c744c07e5e8b95fad5b5746b"
server_key_version: "encryptionkey@0"
I20260812 18:54:06.695844 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "869ed47ea6194f9194abb85fcd53ea45"
format_stamp: "Formatted at 2026-08-12 18:54:06 on dist-test-slave-rnrw"
server_key: "4346c9be1de7ae959e9a64b8d1122abc"
server_key_iv: "cf0ab733c744c07e5e8b95fad5b5746b"
server_key_version: "encryptionkey@0"
I20260812 18:54:06.700220 23741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 18:54:06.703125 23757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:06.704725 23741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:06.704883 23741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "869ed47ea6194f9194abb85fcd53ea45"
format_stamp: "Formatted at 2026-08-12 18:54:06 on dist-test-slave-rnrw"
server_key: "4346c9be1de7ae959e9a64b8d1122abc"
server_key_iv: "cf0ab733c744c07e5e8b95fad5b5746b"
server_key_version: "encryptionkey@0"
I20260812 18:54:06.705024 23741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:06.716162 23741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:06.719805 23741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:06.720036 23741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:06.727972 23741 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:41697
I20260812 18:54:06.728276 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:41697 every 8 connection(s)
I20260812 18:54:06.729193 23741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:06.732340 23818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:06.732776 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23741
I20260812 18:54:06.732929 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:06.733265 16547 external_mini_cluster.cc:1428] Setting key 696ce39437cd84bfb4b04e92fb380096
I20260812 18:54:06.740339 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Bootstrap starting.
Aug 12 18:54:06 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:06.743392 23818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:06.744199 23818 log.cc:824] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:06.746289 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: No bootstrap required, opened a new log
I20260812 18:54:06.748612 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:06.734867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.16.40.254:41697)
0812 18:54:06.735362 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:06.735380 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:06.735421 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:54:06.736165 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:06.736186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:06.736621 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:06.736948 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:06.737943 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:06.738959 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:06.740144 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:06.740756 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:06.740883 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:06.743817 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:06.743823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:06.743839 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:06.743868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:06.745703 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:06.746173 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:06.746182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:06.746192 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:06.746304 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:06.746604 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:06.746607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:06.746609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:06.747009 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:06.747219 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:06.747535 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 18:54:06.747839 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":165,"threads_started":1}
I20260812 18:54:06.750514 23818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } }
I20260812 18:54:06.750795 23818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:06.750856 23818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869ed47ea6194f9194abb85fcd53ea45, State: Initialized, Role: FOLLOWER
I20260812 18:54:06.751322 23818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } }
I20260812 18:54:06.751451 23818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:06.751498 23818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:06.751573 23818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:06.752485 23818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } }
I20260812 18:54:06.752838 23818 leader_election.cc:302] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869ed47ea6194f9194abb85fcd53ea45; no voters:
I20260812 18:54:06.753118 23818 leader_election.cc:288] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:06.753295 23826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:06.753543 23826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 LEADER]: Becoming Leader. State: Replica: 869ed47ea6194f9194abb85fcd53ea45, State: Running, Role: LEADER
I20260812 18:54:06.753894 23826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } }
I20260812 18:54:06.754455 23818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:06.755654 23828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } }
I20260812 18:54:06.755806 23828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:06.756235 23837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:06.756184 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869ed47ea6194f9194abb85fcd53ea45. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } }
I20260812 18:54:06.756346 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:06.759501 23837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:06.766016 23837 catalog_manager.cc:1347] Generated new cluster ID: fadcf02855ba43fdb8c82bfb05a0ea83
I20260812 18:54:06.766155 23837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:06.774773 23837 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:06.776167 23837 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:06.791862 23837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Generated new TSK 0
I20260812 18:54:06.792995 23837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
I20260812 18:54:06.833930 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:06.988471 23853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:06.988857 23853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:06.988910 23853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:06.993254 23853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:06.993314 23853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:06.993373 23853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:06.998678 23853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:06.999689 23853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:07.000775 23853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:07.010573 23862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.010601 23858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.010566 23860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.010792 23853 server_base.cc:1034] running on GCE node
I20260812 18:54:07.011292 23853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:07.011988 23853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:07.013198 23853 hybrid_clock.cc:648] HybridClock initialized: now 1786560847013164 us; error 28 us; skew 500 ppm
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:07.015971 23853 init.cc:377] Logged in from keytab as oryx/127.16.40.193@KRBTEST.COM (short username oryx)
I20260812 18:54:07.016947 23853 webserver.cc:460] Webserver started at http://127.16.40.193:42461/ using document root <none> and password file <none>
I20260812 18:54:07.017537 23853 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.017632 23853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.017874 23853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:07.019896 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "e33f3265a9ee42abb43f61a8bfb5175a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "212a4395e37a17f2c5cfc8025e417999"
server_key_iv: "2a9c6e1a791422e4a152feb3c6cd8be1"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.020480 23853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "e33f3265a9ee42abb43f61a8bfb5175a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "212a4395e37a17f2c5cfc8025e417999"
server_key_iv: "2a9c6e1a791422e4a152feb3c6cd8be1"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.024415 23853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:07.027130 23870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.028533 23853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:07.028793 23853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "e33f3265a9ee42abb43f61a8bfb5175a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "212a4395e37a17f2c5cfc8025e417999"
server_key_iv: "2a9c6e1a791422e4a152feb3c6cd8be1"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.028947 23853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:07.039770 23853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:07.042914 23853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:07.043150 23853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:07.043793 23853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:07.044725 23853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:07.044801 23853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.044878 23853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:07.044925 23853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.054461 23853 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43461
I20260812 18:54:07.054562 24001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43461 every 8 connection(s)
I20260812 18:54:07.055532 23853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:07.062887 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 23853
I20260812 18:54:07.063041 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:07.063346 16547 external_mini_cluster.cc:1428] Setting key 0b0069bfc9503dd8efe5e228746b53b3
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:07.070863 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.057570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:42881 (local address 127.16.40.254:41697)
0812 18:54:07.057727 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.057731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.058781 (+ 1050us) server_negotiation.cc:407] Connection header received
0812 18:54:07.059694 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.059697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.059758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.059837 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.061094 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.061642 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.062896 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.063259 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.063354 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.066305 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.066311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.066319 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.066344 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.068222 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.068954 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.068961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.068966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.069090 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.069454 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.069461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.069465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.069745 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.069859 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.070467 (+ 608us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.070640 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:07.071661 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.057945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:42881)
0812 18:54:07.058629 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.058671 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.059490 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.060017 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.060024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.060379 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.060932 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.060947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.061773 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.061779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.062763 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.062775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.063430 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.063435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.063637 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.064320 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.064336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.066129 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.068369 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.068374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.068387 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.068841 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.069194 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.069197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.069198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.069345 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.069855 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.069861 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.070135 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.070351 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.070664 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":122,"threads_started":1}
I20260812 18:54:07.073120 24005 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:07.073369 24005 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:07.073909 24005 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:54:07.075549 23775 ts_manager.cc:194] Registered new tserver with Master: e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193:43461)
I20260812 18:54:07.076947 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:42881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
I20260812 18:54:07.096138 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:07.212777 24010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:07.213099 24010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:07.213179 24010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:07.217283 24010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:07.217394 24010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:07.217492 24010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:07.223134 24010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:07.224356 24010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:07.225492 24010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:07.236579 24017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.236595 24015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.236578 24019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.236933 24010 server_base.cc:1034] running on GCE node
I20260812 18:54:07.237435 24010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:07.238163 24010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:07.239344 24010 hybrid_clock.cc:648] HybridClock initialized: now 1786560847239321 us; error 36 us; skew 500 ppm
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:07.242388 24010 init.cc:377] Logged in from keytab as oryx/127.16.40.194@KRBTEST.COM (short username oryx)
I20260812 18:54:07.243327 24010 webserver.cc:460] Webserver started at http://127.16.40.194:39793/ using document root <none> and password file <none>
I20260812 18:54:07.243901 24010 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.243965 24010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.244164 24010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:07.246469 24010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "cb40b77756fa4e42a033bdca76a48db5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "b90cd74942b9a8d41f902b0030977cb6"
server_key_iv: "9041a08ee09a35ec903b42da3100b4a4"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.247027 24010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "cb40b77756fa4e42a033bdca76a48db5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "b90cd74942b9a8d41f902b0030977cb6"
server_key_iv: "9041a08ee09a35ec903b42da3100b4a4"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.251128 24010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 18:54:07.253821 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.255508 24010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:07.255648 24010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "cb40b77756fa4e42a033bdca76a48db5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "b90cd74942b9a8d41f902b0030977cb6"
server_key_iv: "9041a08ee09a35ec903b42da3100b4a4"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.255848 24010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:07.284204 24010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:07.288004 24010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:07.288230 24010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:07.288964 24010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:07.289880 24010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:07.289937 24010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.290004 24010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:07.290037 24010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.299994 24010 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:39609
I20260812 18:54:07.300017 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:39609 every 8 connection(s)
I20260812 18:54:07.301051 24010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:07.304443 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24010
I20260812 18:54:07.304605 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:07.304888 16547 external_mini_cluster.cc:1428] Setting key 9326fd63689382fe35ba012a1abd569c
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:54:07.316668 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.302996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:52355 (local address 127.16.40.254:41697)
0812 18:54:07.303171 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.303176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.304063 (+ 887us) server_negotiation.cc:407] Connection header received
0812 18:54:07.305725 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.305729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.305786 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.305867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.307194 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.307750 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.309091 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.309391 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.309535 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.312433 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.312437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.312439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.312467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.314185 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.314865 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.314869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.314871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.314968 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.315349 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.315354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.315356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.315576 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.315691 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.316288 (+ 597us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.316454 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:54:07.317797 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.303400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:52355)
0812 18:54:07.303892 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.303939 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.305504 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.306039 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.306046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.306520 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.307040 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.307054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.307898 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.307902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.308951 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.308961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.309606 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.309611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.309837 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.310512 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.310530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.312219 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.314386 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.314395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.314416 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.314747 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.315068 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.315072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.315074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.315245 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.315691 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.315700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.316030 (+ 330us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.316243 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.316665 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":120,"threads_started":1}
I20260812 18:54:07.319096 24157 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:07.319361 24157 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.195@KRBTEST.COM" created.
I20260812 18:54:07.319965 24157 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
I20260812 18:54:07.321136 23775 ts_manager.cc:194] Registered new tserver with Master: cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:07.321925 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:52355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
I20260812 18:54:07.337011 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:07.456753 24162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:07.457082 24162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:07.457160 24162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:07.461542 24162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:07.461621 24162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:07.461715 24162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:07.467471 24162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:07.468715 24162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:07.469820 24162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:07.480639 24167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.480639 24171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.480962 24169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.481724 24162 server_base.cc:1034] running on GCE node
I20260812 18:54:07.482268 24162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:07.482972 24162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:07.484170 24162 hybrid_clock.cc:648] HybridClock initialized: now 1786560847484133 us; error 55 us; skew 500 ppm
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:07.487689 24162 init.cc:377] Logged in from keytab as oryx/127.16.40.195@KRBTEST.COM (short username oryx)
I20260812 18:54:07.488827 24162 webserver.cc:460] Webserver started at http://127.16.40.195:36951/ using document root <none> and password file <none>
I20260812 18:54:07.489472 24162 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.489569 24162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.489820 24162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:07.492048 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "7a176f8855fb4addba2354a26034574a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "12009c99639d98456bdc192f695f6c11"
server_key_iv: "fac2df9fa6276781feda4b261aa0bde9"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.492758 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "7a176f8855fb4addba2354a26034574a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "12009c99639d98456bdc192f695f6c11"
server_key_iv: "fac2df9fa6276781feda4b261aa0bde9"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.497030 24162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:54:07.499825 24180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.501257 24162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:07.501410 24162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "7a176f8855fb4addba2354a26034574a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "12009c99639d98456bdc192f695f6c11"
server_key_iv: "fac2df9fa6276781feda4b261aa0bde9"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.501536 24162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:07.538838 24162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:07.542354 24162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:07.542582 24162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:07.543251 24162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:07.544186 24162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:07.544260 24162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.544332 24162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:07.544377 24162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.553870 24162 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39195
I20260812 18:54:07.553895 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39195 every 8 connection(s)
I20260812 18:54:07.555018 24162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:07.565263 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24162
I20260812 18:54:07.565421 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:07.565848 16547 external_mini_cluster.cc:1428] Setting key 382ab6b349b7b26f41f633054375463b
I20260812 18:54:07.570631 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.557235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:40675 (local address 127.16.40.254:41697)
0812 18:54:07.557384 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.557388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.558311 (+ 923us) server_negotiation.cc:407] Connection header received
0812 18:54:07.559866 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.559869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.559933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.560036 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.561399 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.561937 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.563134 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.563354 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.563466 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.566590 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.566595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.566597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.566631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.568344 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.568972 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.568976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.568977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.569063 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.569459 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.569462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.569464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.569673 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.569784 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.570357 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.570476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:54:07.571514 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.557592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:40675)
0812 18:54:07.558125 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.558182 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.559642 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.560226 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.560233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.560729 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.561246 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.561260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.562071 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.562075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.562982 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.562993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.563516 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.563520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.563699 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.564315 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.564332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.566417 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.568464 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.568471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.568487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.568857 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.569161 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.569163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.569165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.569321 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.569791 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.569797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.570078 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.570250 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.570543 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":122,"threads_started":1}
I20260812 18:54:07.572856 24311 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:07.573117 24311 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:07.573637 24311 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
I20260812 18:54:07.574678 23775 ts_manager.cc:194] Registered new tserver with Master: 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:07.575282 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:40675
I20260812 18:54:07.580423 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.253.keytab.
Entry for principal oryx/127.16.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.253.keytab.
I20260812 18:54:07.612381 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:41923
--webserver_interface=127.16.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.253.keytab
--principal=oryx/127.16.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41697,127.16.40.253:41923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:07.729982 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:07.730274 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:07.730361 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:07.734623 24320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:07.734719 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:07.734757 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:07.734807 24320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:07.734845 24320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:07.740599 24320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41697,127.16.40.253:41923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.253.keytab
--principal=oryx/127.16.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:41923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
W20260812 18:54:07.740878 24320 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.40.254:41697,127.16.40.253:41923'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 18:54:07.741801 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:07.742902 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:07.752015 24330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.752007 24327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.752007 24326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.752333 24320 server_base.cc:1034] running on GCE node
I20260812 18:54:07.753069 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:07.754289 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:07.755513 24320 hybrid_clock.cc:648] HybridClock initialized: now 1786560847755491 us; error 38 us; skew 500 ppm
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:07.759259 24320 init.cc:377] Logged in from keytab as oryx/127.16.40.253@KRBTEST.COM (short username oryx)
I20260812 18:54:07.760442 24320 webserver.cc:460] Webserver started at http://127.16.40.253:43403/ using document root <none> and password file <none>
I20260812 18:54:07.761188 24320 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.761263 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.761457 24320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:07.763815 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/instance:
uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "acb6d32afa4c47f6773649c8eda83537"
server_key_iv: "ee21525d7cb97eb2eb3d9e29e96e29ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.764421 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal/instance:
uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "acb6d32afa4c47f6773649c8eda83537"
server_key_iv: "ee21525d7cb97eb2eb3d9e29e96e29ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.769053 24320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:07.771950 24338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.773398 24320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:07.773533 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "acb6d32afa4c47f6773649c8eda83537"
server_key_iv: "ee21525d7cb97eb2eb3d9e29e96e29ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.773654 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:07.793362 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:07.796627 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:07.796838 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:07.804013 24320 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.253:41923
I20260812 18:54:07.804315 24398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.253:41923 every 8 connection(s)
I20260812 18:54:07.805061 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
I20260812 18:54:07.808066 24400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:07.810400 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24320
I20260812 18:54:07.812269 24400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.253@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
I20260812 18:54:07.825578 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.814388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.16.40.254:41697)
0812 18:54:07.814520 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.814525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.814863 (+ 338us) server_negotiation.cc:407] Connection header received
0812 18:54:07.815031 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.815034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.815082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.815171 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.815880 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.816649 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.817991 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.818334 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.818426 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.821127 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.821131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.821133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.821159 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.823009 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.823579 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.823584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.823585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.823690 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.824116 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.824120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.824122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.824512 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.824651 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.825211 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.825379 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:07.826448 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.814356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53096)
0812 18:54:07.814745 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.814783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.814919 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.815238 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.815242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.815572 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.815734 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.815743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.816765 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.816768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.817867 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.817896 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.818627 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.818630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.818719 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.819314 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.819331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.820933 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.823123 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.823132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.823139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.823472 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.823790 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.823795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.823800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.823983 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.824673 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.824685 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.824946 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.825136 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.825478 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":4,"thread_start_us":128,"threads_started":1}
I20260812 18:54:07.826454 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.812062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.16.40.253:41923)
0812 18:54:07.812497 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.812516 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.812596 (+ 80us) server_negotiation.cc:407] Connection header received
0812 18:54:07.813410 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.813432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.813822 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.814129 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.815201 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.816464 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.817726 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.818217 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.818412 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.821565 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.821575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.821591 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.821637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.823628 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.824130 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.824136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.824138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.824220 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.824488 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.824492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.824493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.824937 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.825150 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.825406 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.825550 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260812 18:54:07.827958 24400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.253@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
I20260812 18:54:07.836494 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.828616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.16.40.253:41923)
0812 18:54:07.828708 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.828712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.828724 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:54:07.828817 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.828822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.828861 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.828936 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.829292 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.830124 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.830844 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.831068 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.831172 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.833296 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.833300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.833302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.833323 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.834768 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.835198 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.835201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.835202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.835273 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.835516 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.835518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.835519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.835694 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.835773 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.836239 (+ 466us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.836363 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":39}
I20260812 18:54:07.836598 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.828450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60316)
0812 18:54:07.828626 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.828642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.828721 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.828949 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.828951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.829110 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.829190 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.829195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.830228 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.830234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.830734 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.830741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.831171 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.831174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.831265 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.831729 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.831746 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.833163 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.834872 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.834875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.834876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.835108 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.835353 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.835355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.835357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.835438 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.835816 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.835818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.836059 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.836178 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.836305 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 18:54:07.839524 24400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Bootstrap starting.
I20260812 18:54:07.841992 24400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:07.842746 24400 log.cc:824] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:07.844877 24400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: No bootstrap required, opened a new log
I20260812 18:54:07.847991 24400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } }
I20260812 18:54:07.848205 24400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:07.848276 24400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be9bed7261d4c98babdceb0b6b3cae5, State: Initialized, Role: FOLLOWER
I20260812 18:54:07.848774 24400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } }
I20260812 18:54:07.849328 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } } }
I20260812 18:54:07.849452 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:07.850063 24400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 18:54:07.855373 24419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:54:07.855487 24419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:54:07.858326 24320 master_runner.cc:428] Detected that this master 2be9bed7261d4c98babdceb0b6b3cae5 is joining an existing cluster
I20260812 18:54:07.858402 24320 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.40.253:41923
I20260812 18:54:07.869915 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.860830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53112)
0812 18:54:07.861230 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.861247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.861384 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.861771 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.861774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.862073 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.862175 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.862181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.863287 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.863290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.863949 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.863958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.865043 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.865046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.865106 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.865695 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.865707 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.866058 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.868059 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.868065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.868381 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.868717 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.868725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.868843 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.869295 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.869298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.869419 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.869530 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.869696 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":53,"spinlock_wait_cycles":31616,"thread_start_us":148,"threads_started":1}
I20260812 18:54:07.869916 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.860881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.16.40.254:41697)
0812 18:54:07.861042 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.861048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.861330 (+ 282us) server_negotiation.cc:407] Connection header received
0812 18:54:07.861507 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.861512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.861573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.861661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.862302 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.863151 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.864076 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.864313 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.864429 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.866204 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.866207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.866210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.866242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.867926 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868499 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.868503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.868505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.868625 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868944 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.868947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.868949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.869192 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.869301 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.869606 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.869733 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:54:07.870429 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.861296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.16.40.253:41923)
0812 18:54:07.861417 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.861423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.861628 (+ 205us) server_negotiation.cc:407] Connection header received
0812 18:54:07.861738 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.861743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.861781 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.861855 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.862423 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.863142 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.864395 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.864738 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.864863 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.866193 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.866197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.866200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.866223 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.868080 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868594 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.868599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.868601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.868702 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868997 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.869002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.869004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.869191 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.869301 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.870158 (+ 857us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.870261 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 18:54:07.869973 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.860878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60324)
0812 18:54:07.861526 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.861537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.861657 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.861899 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.861902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.862169 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.862301 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.862308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.863276 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.863280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.864263 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.864274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.864890 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.864894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.864993 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.865524 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.865539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.866059 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.868181 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.868186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.868481 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.868786 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.868790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.868792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.868888 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.869338 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.869341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.869428 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.869543 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.869774 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":119,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.254@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.40.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.252@KRBTEST.COM" created.
I20260812 18:54:07.894619 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.880528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.16.40.253:41923)
0812 18:54:07.882603 (+ 2075us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.882609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.882627 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:54:07.882717 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.882723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.882791 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.882882 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.884071 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.884899 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.886454 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.886797 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.886913 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.890114 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.890120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.890126 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.890167 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.892093 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.892803 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.892806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.892808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.892896 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.893266 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.893273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.893277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.893573 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.893661 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.894187 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.894349 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1964}
I20260812 18:54:07.896678 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.880516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60330)
0812 18:54:07.881007 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.881081 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.881220 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.883108 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.883118 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.883713 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.883905 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.883929 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.885046 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.885053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.886248 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.886265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.886945 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.886952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.887114 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.887808 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.887830 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.889899 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.892231 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.892246 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.892254 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.892677 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.893009 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.893013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.893015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.893127 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.893680 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.893688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.893847 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.894014 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.894312 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260812 18:54:07.898684 23775 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.40.253:41923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
member_type: NON_VOTER
last_known_addr {
host: "127.16.40.253"
port: 41923
}
attrs {
promote: true
}
}
dest_uuid: "869ed47ea6194f9194abb85fcd53ea45"
cas_config_opid_index: -1
I20260812 18:54:07.899506 23775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } }
I20260812 18:54:07.901886 24436 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:07.902706 24369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:07.904323 24369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 18:54:07.905234 24437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:07.907624 24436 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.40.253:41923
I20260812 18:54:07.908003 24437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:07.908142 24437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:07.908245 24320 master.cc:537] Master@127.16.40.253:41923 shutting down...
I20260812 18:54:07.914752 24369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.252.keytab.
Entry for principal oryx/127.16.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.252.keytab.
I20260812 18:54:07.916299 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:37141
--webserver_interface=127.16.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.252.keytab
--principal=oryx/127.16.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:54:07.917496 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869ed47ea6194f9194abb85fcd53ea45. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:07.917634 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:07.917734 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:07.917801 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:07.917860 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:07.917928 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:07.919247 24440 mvcc.cc:205] Tried to move back new op lower bound from 7317753228437430272 to 7317753228304941056. Current Snapshot: MvccSnapshot[applied={T|T < 7317753228342095872}]
I20260812 18:54:07.920218 24320 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Raft consensus shutting down.
I20260812 18:54:07.920581 24320 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Raft consensus is shut down!
I20260812 18:54:07.920682 24320 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: stopping tablet replica
I20260812 18:54:07.932745 24320 master.cc:552] Master@127.16.40.253:41923 shutdown complete.
I20260812 18:54:07.932888 24320 master_runner.cc:305] Clearing existing system tablet
I20260812 18:54:07.933277 24320 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.933403 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.934872 24320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:07.936148 24451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.936388 24320 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:07.936507 24320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:07.936578 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "acb6d32afa4c47f6773649c8eda83537"
server_key_iv: "ee21525d7cb97eb2eb3d9e29e96e29ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.938715 24320 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:07.941447 24320 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 18:54:07.941567 24320 log.cc:1196] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 18:54:07.941895 24320 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Deleting consensus metadata
I20260812 18:54:07.942092 24320 master_runner.cc:315] Copying system tablet from 127.16.40.254:41697
I20260812 18:54:07.943552 24320 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.40.254:41697
I20260812 18:54:07.954304 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.944380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.16.40.254:41697)
0812 18:54:07.944516 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:07.944521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:07.944807 (+ 286us) server_negotiation.cc:407] Connection header received
0812 18:54:07.944933 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:07.944936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:07.944997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:07.945091 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:07.945638 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.946438 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.948166 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.948629 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.948751 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.950150 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:07.950154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:07.950157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:07.950188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:07.952312 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.952863 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.952868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.952869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.952966 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:07.953268 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:07.953271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:07.953273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:07.953530 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:07.953631 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:07.954010 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 18:54:07.954138 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:54:07.954339 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:07.944267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53118)
0812 18:54:07.944692 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:07.944707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:07.944838 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:07.945125 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:07.945128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:07.945361 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:07.945506 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.945513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.946566 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.946570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.947539 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:07.947555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:07.948762 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:07.948767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:07.948875 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:07.949483 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:07.949499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:07.949953 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:07.952429 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.952433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.952436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.952742 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.953068 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:07.953071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:07.953073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:07.953171 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:07.953645 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:07.953650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:07.953799 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 18:54:07.953942 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 18:54:07.954126 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":102,"threads_started":1}
I20260812 18:54:07.955461 23798 tablet_copy_service.cc:140] P 869ed47ea6194f9194abb85fcd53ea45: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2be9bed7261d4c98babdceb0b6b3cae5 ({username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:53118)
I20260812 18:54:07.955659 23798 tablet_copy_service.cc:161] P 869ed47ea6194f9194abb85fcd53ea45: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2be9bed7261d4c98babdceb0b6b3cae5 at {username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:53118: session id = 2be9bed7261d4c98babdceb0b6b3cae5-00000000000000000000000000000000
I20260812 18:54:07.957664 23798 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 18:54:07.959836 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 18:54:07.962651 24320 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: tablet copy: Starting download of 0 data blocks...
I20260812 18:54:07.962980 24320 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: tablet copy: Starting download of 1 WAL segments...
I20260812 18:54:07.965006 24320 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 18:54:07.967103 23798 tablet_copy_service.cc:342] P 869ed47ea6194f9194abb85fcd53ea45: Request end of tablet copy session 2be9bed7261d4c98babdceb0b6b3cae5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:53118
I20260812 18:54:07.967253 23798 tablet_copy_service.cc:433] P 869ed47ea6194f9194abb85fcd53ea45: ending tablet copy session 2be9bed7261d4c98babdceb0b6b3cae5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2be9bed7261d4c98babdceb0b6b3cae5
I20260812 18:54:07.976263 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:07.978220 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:07.978271 24459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.978816 24320 server_base.cc:1034] running on GCE node
W20260812 18:54:07.978878 24461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:07.979251 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:07.979573 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:07.980703 24320 hybrid_clock.cc:648] HybridClock initialized: now 1786560847980692 us; error 31 us; skew 500 ppm
Aug 12 18:54:07 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560847, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:07.982543 24320 init.cc:377] Logged in from keytab as oryx/127.16.40.253@KRBTEST.COM (short username oryx)
I20260812 18:54:07.983107 24320 webserver.cc:460] Webserver started at http://127.16.40.253:38879/ using document root <none> and password file <none>
I20260812 18:54:07.983312 24320 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:07.983363 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:07.984862 24320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:07.985980 24468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:07.986267 24320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:07.986349 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "acb6d32afa4c47f6773649c8eda83537"
server_key_iv: "ee21525d7cb97eb2eb3d9e29e96e29ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:07.986438 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.000383 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.002774 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.008723 24320 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.253:41923
I20260812 18:54:08.009477 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
I20260812 18:54:08.009922 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.253:41923 every 8 connection(s)
I20260812 18:54:08.011620 24535 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:54:08.012862 24535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Bootstrap starting.
I20260812 18:54:08.021416 24535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:54:08.021896 24535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Bootstrap complete.
I20260812 18:54:08.022617 24535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } }
I20260812 18:54:08.022867 24535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2be9bed7261d4c98babdceb0b6b3cae5, State: Initialized, Role: LEARNER
I20260812 18:54:08.023033 24535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } }
I20260812 18:54:08.023555 24538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } }
I20260812 18:54:08.023703 24538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:08.023887 24535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:08.027889 24549 catalog_manager.cc:1259] Loaded cluster ID: fadcf02855ba43fdb8c82bfb05a0ea83
I20260812 18:54:08.027961 24549 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: loading cluster ID for follower catalog manager: success
I20260812 18:54:08.030112 24549 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: acquiring CA information for follower catalog manager: success
I20260812 18:54:08.031592 24549 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 18:54:08.065188 24445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:08.065495 24445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:08.065536 24445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:08.069741 24445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:08.069820 24445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:08.069841 24445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:08.069859 24445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:08.069873 24445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:08.075562 24445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.252.keytab
--principal=oryx/127.16.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:08.076782 24445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:08.077844 24445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:08.079833 24005 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
W20260812 18:54:08.087471 24557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.087450 24555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.087532 24554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.087746 24445 server_base.cc:1034] running on GCE node
I20260812 18:54:08.088470 24445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:08.089824 24445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:08.090997 24445 hybrid_clock.cc:648] HybridClock initialized: now 1786560848090966 us; error 45 us; skew 500 ppm
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:08.094446 24445 init.cc:377] Logged in from keytab as oryx/127.16.40.252@KRBTEST.COM (short username oryx)
I20260812 18:54:08.095459 24445 webserver.cc:460] Webserver started at http://127.16.40.252:32853/ using document root <none> and password file <none>
I20260812 18:54:08.096025 24445 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.096091 24445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.096267 24445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:08.098431 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/instance:
uuid: "7844627255b44fa98e1f07504f7c87a8"
format_stamp: "Formatted at 2026-08-12 18:54:08 on dist-test-slave-rnrw"
server_key: "d49a51b5230211430efd3947868e400b"
server_key_iv: "625e76666e8b6530d20ceba654d20c05"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.098973 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal/instance:
uuid: "7844627255b44fa98e1f07504f7c87a8"
format_stamp: "Formatted at 2026-08-12 18:54:08 on dist-test-slave-rnrw"
server_key: "d49a51b5230211430efd3947868e400b"
server_key_iv: "625e76666e8b6530d20ceba654d20c05"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.102996 24445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:54:08.105684 24566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.107100 24445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:08.107255 24445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "7844627255b44fa98e1f07504f7c87a8"
format_stamp: "Formatted at 2026-08-12 18:54:08 on dist-test-slave-rnrw"
server_key: "d49a51b5230211430efd3947868e400b"
server_key_iv: "625e76666e8b6530d20ceba654d20c05"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.107379 24445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.119823 24445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.123178 24445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:08.123400 24445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.132201 24445 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.252:37141
I20260812 18:54:08.132707 24624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.252:37141 every 8 connection(s)
I20260812 18:54:08.133437 24445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
I20260812 18:54:08.136101 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24445
I20260812 18:54:08.137362 24625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:08.142021 24625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.252@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:08.152733 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.137945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.16.40.252:37141)
0812 18:54:08.138428 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.138443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.138482 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:54:08.139145 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.139161 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.139491 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.139864 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.140758 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.141785 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.142712 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.143194 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.143360 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.146962 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.146968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.146979 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.147009 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.149187 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.149670 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.149677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.149684 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.149843 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.150173 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.150177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.150178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.150713 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.150989 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.151245 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.151667 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":166,"threads_started":1}
I20260812 18:54:08.154155 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.144069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.16.40.254:41697)
0812 18:54:08.144250 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.144256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.144603 (+ 347us) server_negotiation.cc:407] Connection header received
0812 18:54:08.144834 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.144837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.144887 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.144976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.145928 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.146688 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.147662 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.147889 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.147999 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.150499 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.150503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.150505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.150530 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.152088 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.152667 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.152669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.152670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.152748 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.153210 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.153213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.153215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.153442 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.153556 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.153905 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.154022 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 18:54:08.154211 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.143939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53126)
0812 18:54:08.144449 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.144506 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.144710 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.145066 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.145073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.145546 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.145688 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.145694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.146842 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.146849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.147527 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.147544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.148002 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.148006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.148131 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.148706 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.148718 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.150357 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.152205 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.152213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.152219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.152529 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.152980 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.152984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.152986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.153090 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.153577 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.153582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.153686 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.153802 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.154030 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":193,"threads_started":1}
I20260812 18:54:08.155377 24625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.252@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
I20260812 18:54:08.164821 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.156032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.16.40.253:41923)
0812 18:54:08.156293 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.156296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.156307 (+ 11us) server_negotiation.cc:407] Connection header received
0812 18:54:08.156342 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.156344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.156382 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.156460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.156994 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.157692 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.158466 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.158732 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.158845 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.161053 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.161057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.161060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.161088 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.162805 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.163334 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.163338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.163340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.163435 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.163743 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.163748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.163751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.163977 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.164097 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.164411 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.164597 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260812 18:54:08.164808 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.155907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60334)
0812 18:54:08.156012 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.156026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.156094 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.156499 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.156501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.156719 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.156854 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.156864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.157786 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.157789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.158321 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.158328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.158846 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.158848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.158924 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.159380 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.159392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.160877 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.162978 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.162982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.162984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.163223 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.163541 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.163545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.163548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.163643 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.164085 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.164098 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.164218 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.164335 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.164521 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 18:54:08.165813 24625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.252@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
I20260812 18:54:08.174281 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.166396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:53498)
0812 18:54:08.166513 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.166526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.166601 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.166814 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.166816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.166987 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.167074 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.167079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.168205 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.168208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.168879 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.168889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.169333 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.169335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.169395 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.169947 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.169963 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.171178 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.172846 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.172849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.172850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.173070 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.173323 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.173326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.173327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.173414 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.173798 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.173801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.173889 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.173983 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.174114 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 18:54:08.174301 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.166524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.16.40.252:37141)
0812 18:54:08.166638 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.166641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.166653 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:54:08.166681 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.166683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.166718 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.166783 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.167173 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.167689 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.168992 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.169233 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.169316 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.171307 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.171310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.171312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.171332 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.172729 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.173167 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.173169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.173171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.173242 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.173541 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.173543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.173544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.173716 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.173790 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.174031 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.174172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":44}
I20260812 18:54:08.177795 24625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Bootstrap starting.
I20260812 18:54:08.180260 24625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:08.181159 24625 log.cc:824] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:08.183244 24625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: No bootstrap required, opened a new log
I20260812 18:54:08.186573 24625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } }
I20260812 18:54:08.186864 24625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:08.186923 24625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7844627255b44fa98e1f07504f7c87a8, State: Initialized, Role: FOLLOWER
I20260812 18:54:08.187366 24625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } }
I20260812 18:54:08.188005 24631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } } }
I20260812 18:54:08.188153 24631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:08.188814 24625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 18:54:08.194114 24643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:54:08.194236 24643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:54:08.197924 24445 master_runner.cc:428] Detected that this master 7844627255b44fa98e1f07504f7c87a8 is joining an existing cluster
I20260812 18:54:08.198011 24445 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.40.252:37141
I20260812 18:54:08.207592 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23853
I20260812 18:54:08.213243 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.200747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.16.40.254:41697)
0812 18:54:08.200881 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.200887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.201705 (+ 818us) server_negotiation.cc:407] Connection header received
0812 18:54:08.201903 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.201909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.201968 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.202025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.202820 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203609 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205090 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.205455 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205520 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.206956 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.206960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.206963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.206997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.211051 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.211779 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.211783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.211785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.211881 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.212196 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.212201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.212202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.212474 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.212657 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.212946 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.213066 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:54:08.213265 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.200652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53136)
0812 18:54:08.201613 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.201626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.201776 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.202187 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.202189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.202516 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.202676 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.202685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203738 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.203743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.204948 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.204959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.205691 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.205770 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.206342 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.206357 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.206818 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.211360 (+ 4542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.211363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.211366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.211657 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.211979 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.211982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.211985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.212087 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.212640 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.212643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.212773 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.212885 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.213065 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":207,"spinlock_wait_cycles":896,"thread_start_us":96,"threads_started":1}
I20260812 18:54:08.228965 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:43461
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:41697
--webserver_port=42461
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:54:08.229441 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.201181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.16.40.252:37141)
0812 18:54:08.201420 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.201426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.201924 (+ 498us) server_negotiation.cc:407] Connection header received
0812 18:54:08.202050 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.202053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.202098 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.202148 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.202800 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203539 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205077 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.205382 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205453 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.225827 (+ 20374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.225831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.225834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.225858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.227600 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.228095 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.228099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.228101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.228186 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.228439 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.228442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.228443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.228751 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.228826 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.229138 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.229274 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":185}
I20260812 18:54:08.229970 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.201514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:53502)
0812 18:54:08.201855 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.201870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.201966 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.202304 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.202309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.202558 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.202696 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.202702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203677 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.203683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.204948 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.204959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.205646 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.205743 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.225041 (+ 19298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.225058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.225636 (+ 578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.227722 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.227726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.227729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.227995 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.228283 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.228286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.228288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.228358 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.228885 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.228887 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.228967 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.229667 (+ 700us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.229806 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":159,"thread_start_us":74,"threads_started":1}
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560846, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.254@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
I20260812 18:54:08.233848 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.201150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.16.40.253:41923)
0812 18:54:08.201319 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.201325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.201346 (+ 21us) server_negotiation.cc:407] Connection header received
0812 18:54:08.201524 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.201528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.201589 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.201685 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.202418 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203251 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205215 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.205599 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.205677 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.227290 (+ 21613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.227298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.227304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.227346 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.229678 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.232290 (+ 2612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.232294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.232296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.232401 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.232843 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.232849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.232853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.233084 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.233183 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.233539 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.233689 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:54:08.234360 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.200640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60348)
0812 18:54:08.201279 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.201295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.201398 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.201823 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.201827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.202080 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.202268 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.202277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.203402 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.203406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.205011 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.205023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.206015 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.206019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.206122 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.226555 (+ 20433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.226575 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.227074 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.231906 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.231909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.231911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.232165 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.232518 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.232522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.232524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.232720 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.233243 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.233246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.233352 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.234016 (+ 664us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.234184 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":27,"thread_start_us":125,"threads_started":1}
I20260812 18:54:08.237287 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.221956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:53504)
0812 18:54:08.222134 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.222151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.222243 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.222847 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.222853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.223122 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.223288 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.223300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.224494 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.224501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.225623 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.225640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.227807 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.227814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.227948 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.230005 (+ 2057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.230027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.231661 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.235123 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.235127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.235130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.235409 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.235740 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.235746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.235751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.235872 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.236512 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.236519 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.236710 (+ 191us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.236817 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.237020 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 18:54:08.238178 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.222237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.16.40.252:37141)
0812 18:54:08.222533 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.222542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.222555 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:08.222627 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.222630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.222700 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.222795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.223427 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.224381 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.225955 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.226400 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.232322 (+ 5922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.232465 (+ 143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.232470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.232477 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.232500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.234981 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.235534 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.235538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.235540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.235623 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.236009 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.236014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.236018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.236366 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.237638 (+ 1272us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.237747 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.237908 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":115,"threads_started":1}
I20260812 18:54:08.239300 23775 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.40.252:37141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7844627255b44fa98e1f07504f7c87a8"
member_type: NON_VOTER
last_known_addr {
host: "127.16.40.252"
port: 37141
}
attrs {
promote: true
}
}
dest_uuid: "869ed47ea6194f9194abb85fcd53ea45"
cas_config_opid_index: 5
I20260812 18:54:08.240142 23775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } }
I20260812 18:54:08.241504 24437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.242566 24596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:08.243180 24437 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.40.252:37141
I20260812 18:54:08.243353 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.243480 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.244365 24596 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 18:54:08.245105 24445 master.cc:537] Master@127.16.40.252:37141 shutting down...
I20260812 18:54:08.247479 24438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 18:54:08.248987 23761 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 18:54:08.250051 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.241781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:60360)
0812 18:54:08.244771 (+ 2990us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.244787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.244886 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.245413 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.245416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.245436 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:08.245535 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.245541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.246787 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.246790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.248323 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.248336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.249601 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.249605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.249702 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.249754 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.249831 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.249913 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2906}
I20260812 18:54:08.250535 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.242028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.16.40.253:41923)
0812 18:54:08.242176 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.242184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.244980 (+ 2796us) server_negotiation.cc:407] Connection header received
0812 18:54:08.245040 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.245043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.245118 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.245210 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:08.245685 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.246561 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.248481 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.249187 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.249293 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.249473 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.250322 (+ 849us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.250419 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:08.251487 24496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 18:54:08.252034 24438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 18:54:08.255072 24496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.257189 24538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869ed47ea6194f9194abb85fcd53ea45. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.257316 24538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:08.257408 24538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: NON_VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: true } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.257488 24538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:08.258071 24436 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: attempting to promote NON_VOTER 2be9bed7261d4c98babdceb0b6b3cae5 to VOTER
I20260812 18:54:08.258424 24445 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:08.258648 24436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } }
I20260812 18:54:08.258759 24445 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:08.258867 24445 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: stopping tablet replica
I20260812 18:54:08.260353 24496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 LEARNER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 18:54:08.260804 24437 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 18:54:08.262737 24438 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.263326 24496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.264469 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.264658 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.264816 24438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.264915 24438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.265492 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.265154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:53508)
0812 18:54:08.265323 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.265407 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 18:54:08.265766 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.265918 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:08.266578 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.266330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:53514)
0812 18:54:08.266469 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.266513 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 18:54:08.266721 23761 consensus_peers.cc:487] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 -> Peer 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252:37141): Couldn't send request to peer 7844627255b44fa98e1f07504f7c87a8. Status: Network error: Client connection negotiation failed: client connection to 127.16.40.252:37141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 18:54:08.267674 24445 master.cc:552] Master@127.16.40.252:37141 shutdown complete.
I20260812 18:54:08.267732 24445 master_runner.cc:305] Clearing existing system tablet
I20260812 18:54:08.267997 24445 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.268059 24445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.269449 24445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 18:54:08.270414 24663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.270679 24445 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.270973 24445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:08.271070 24445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "7844627255b44fa98e1f07504f7c87a8"
format_stamp: "Formatted at 2026-08-12 18:54:08 on dist-test-slave-rnrw"
server_key: "d49a51b5230211430efd3947868e400b"
server_key_iv: "625e76666e8b6530d20ceba654d20c05"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.273370 24445 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:08.275978 24445 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 18:54:08.276183 24445 log.cc:1196] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 18:54:08.276521 24445 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Deleting consensus metadata
I20260812 18:54:08.276794 24445 master_runner.cc:315] Copying system tablet from 127.16.40.254:41697
I20260812 18:54:08.278118 24445 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.40.254:41697
I20260812 18:54:08.288250 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.278795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:53152)
0812 18:54:08.279135 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.279152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.279280 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.279584 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.279587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.279831 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.279960 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.279967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.281162 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.281166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.282115 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.282127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.282735 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.282739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.282835 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.283492 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.283508 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.283998 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.286281 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.286285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.286288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.286568 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.286884 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.286887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.286889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.286994 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.287626 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.287628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.287734 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.287846 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.288020 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":96,"threads_started":1}
I20260812 18:54:08.288275 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.278892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.16.40.254:41697)
0812 18:54:08.279031 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.279036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.279245 (+ 209us) server_negotiation.cc:407] Connection header received
0812 18:54:08.279374 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.279378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.279435 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.279547 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.280099 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.281015 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.282256 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.282591 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.282702 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.284158 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.284162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.284166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.284199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.286161 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.286680 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.286684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.286686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.286782 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.287121 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.287125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.287127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.287465 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.287545 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.287943 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.288105 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:08.289552 23798 tablet_copy_service.cc:140] P 869ed47ea6194f9194abb85fcd53ea45: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7844627255b44fa98e1f07504f7c87a8 ({username='oryx', principal='oryx/127.16.40.252@KRBTEST.COM'} at 127.0.0.1:53152)
I20260812 18:54:08.289670 23798 tablet_copy_service.cc:161] P 869ed47ea6194f9194abb85fcd53ea45: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7844627255b44fa98e1f07504f7c87a8 at {username='oryx', principal='oryx/127.16.40.252@KRBTEST.COM'} at 127.0.0.1:53152: session id = 7844627255b44fa98e1f07504f7c87a8-00000000000000000000000000000000
I20260812 18:54:08.291060 23798 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 18:54:08.292476 24445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 18:54:08.295423 24445 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: tablet copy: Starting download of 0 data blocks...
I20260812 18:54:08.295763 24445 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: tablet copy: Starting download of 1 WAL segments...
I20260812 18:54:08.297905 24445 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 18:54:08.300287 23798 tablet_copy_service.cc:342] P 869ed47ea6194f9194abb85fcd53ea45: Request end of tablet copy session 7844627255b44fa98e1f07504f7c87a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.40.252@KRBTEST.COM'} at 127.0.0.1:53152
I20260812 18:54:08.300401 23798 tablet_copy_service.cc:433] P 869ed47ea6194f9194abb85fcd53ea45: ending tablet copy session 7844627255b44fa98e1f07504f7c87a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7844627255b44fa98e1f07504f7c87a8
I20260812 18:54:08.311918 24445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:08.313813 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.313822 24679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.314448 24675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.314875 24445 server_base.cc:1034] running on GCE node
I20260812 18:54:08.315073 24445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:08.315390 24445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:08.316534 24445 hybrid_clock.cc:648] HybridClock initialized: now 1786560848316513 us; error 39 us; skew 500 ppm
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:08.318194 24445 init.cc:377] Logged in from keytab as oryx/127.16.40.252@KRBTEST.COM (short username oryx)
I20260812 18:54:08.318795 24445 webserver.cc:460] Webserver started at http://127.16.40.252:44261/ using document root <none> and password file <none>
I20260812 18:54:08.319029 24445 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.319090 24445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.320650 24445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 18:54:08.321732 24687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.321995 24445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:08.322117 24445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "7844627255b44fa98e1f07504f7c87a8"
format_stamp: "Formatted at 2026-08-12 18:54:08 on dist-test-slave-rnrw"
server_key: "d49a51b5230211430efd3947868e400b"
server_key_iv: "625e76666e8b6530d20ceba654d20c05"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.322273 24445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.324210 24157 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
I20260812 18:54:08.344381 24445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.346931 24445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.352921 24445 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.252:37141
I20260812 18:54:08.352999 24746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.252:37141 every 8 connection(s)
I20260812 18:54:08.353950 24445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
I20260812 18:54:08.356252 24747 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:54:08.357445 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Bootstrap starting.
I20260812 18:54:08.370569 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:54:08.370965 24747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: Bootstrap complete.
I20260812 18:54:08.371634 24747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } }
I20260812 18:54:08.371791 24747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7844627255b44fa98e1f07504f7c87a8, State: Initialized, Role: LEARNER
I20260812 18:54:08.371928 24747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } }
I20260812 18:54:08.372318 24752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.372447 24747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:08.372534 24752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: LEARNER
I20260812 18:54:08.375834 24766 catalog_manager.cc:1259] Loaded cluster ID: fadcf02855ba43fdb8c82bfb05a0ea83
I20260812 18:54:08.375965 24766 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: loading cluster ID for follower catalog manager: success
I20260812 18:54:08.378798 24766 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: acquiring CA information for follower catalog manager: success
I20260812 18:54:08.380342 24766 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 18:54:08.392045 24656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:08.392346 24656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:08.392392 24656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:08.396700 24656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:08.396776 24656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:08.396867 24656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:08.402347 24656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:43461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=42461
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:08.403573 24656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:08.404739 24656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:08.415108 24776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.415167 24772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.415112 24778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.415464 24656 server_base.cc:1034] running on GCE node
I20260812 18:54:08.415980 24656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:08.416698 24656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:08.417876 24656 hybrid_clock.cc:648] HybridClock initialized: now 1786560848417840 us; error 50 us; skew 500 ppm
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:08.420711 24656 init.cc:377] Logged in from keytab as oryx/127.16.40.193@KRBTEST.COM (short username oryx)
I20260812 18:54:08.421666 24656 webserver.cc:460] Webserver started at http://127.16.40.193:42461/ using document root <none> and password file <none>
I20260812 18:54:08.422272 24656 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.422391 24656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.426929 24656 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:08.429319 24786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.430811 24656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:08.430932 24656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "e33f3265a9ee42abb43f61a8bfb5175a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "212a4395e37a17f2c5cfc8025e417999"
server_key_iv: "2a9c6e1a791422e4a152feb3c6cd8be1"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.431524 24656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.445849 24656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.449151 24656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:08.449334 24656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.449930 24656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:08.450917 24656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:08.450994 24656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.451061 24656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:08.451104 24656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.461337 24656 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43461
I20260812 18:54:08.461390 24913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43461 every 8 connection(s)
I20260812 18:54:08.462508 24656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:08.470644 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24656
I20260812 18:54:08.470887 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24010
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:08.481489 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.464643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:34661 (local address 127.16.40.253:41923)
0812 18:54:08.464789 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.464797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.466103 (+ 1306us) server_negotiation.cc:407] Connection header received
0812 18:54:08.467588 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.467594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.467654 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.467742 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.469251 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.469787 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.471716 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.472216 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.472324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.476726 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.476733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.476738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.476774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.478592 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.479253 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.479257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.479259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.479405 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.479906 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.479911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.479915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.480178 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.480282 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.481120 (+ 838us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.481258 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:54:08.482985 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.464632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:46807 (local address 127.16.40.252:37141)
0812 18:54:08.464907 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.464910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.466100 (+ 1190us) server_negotiation.cc:407] Connection header received
0812 18:54:08.467414 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.467418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.467485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.467591 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.469273 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.470355 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.471903 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.472278 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.472358 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.476770 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.476779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.476785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.476830 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.479834 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.480667 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.480674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.480679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.480804 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.481242 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.481249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.481254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.481577 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.481688 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.482498 (+ 810us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.482691 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260812 18:54:08.483407 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.465264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.16.40.193:34661)
0812 18:54:08.465913 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.465980 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.467140 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.467788 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.467800 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.468409 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.469092 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.469107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.469937 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.469940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.471544 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.471560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.472640 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.472646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.472943 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.473911 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.473937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.476479 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.478720 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.478728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.478749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.479129 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.479547 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.479552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.479556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.479778 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.480355 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.480362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.480791 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.481674 (+ 883us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.482087 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":97,"spinlock_wait_cycles":12544,"thread_start_us":165,"threads_started":1}
I20260812 18:54:08.484773 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.465264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.16.40.193:46807)
0812 18:54:08.465913 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.465980 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.467140 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.467787 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.467809 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.468407 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.469102 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.469111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.470505 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.470509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.471768 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.471782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.472641 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.472648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.472949 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.474549 (+ 1600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.474578 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.476505 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.480015 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.480022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.480027 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.480485 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.480950 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.480956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.480961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.481105 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.482081 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.482087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.482231 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.484155 (+ 1924us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.484384 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":173,"spinlock_wait_cycles":54912,"thread_start_us":178,"threads_started":1}
I20260812 18:54:08.486171 24914 heartbeater.cc:347] Connected to a master server at 127.16.40.252:37141
I20260812 18:54:08.486404 24916 heartbeater.cc:347] Connected to a master server at 127.16.40.253:41923
I20260812 18:54:08.486483 24916 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.486482 24914 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.487214 24916 heartbeater.cc:511] Master 127.16.40.253:41923 requested a full tablet report, sending...
I20260812 18:54:08.487267 24914 heartbeater.cc:511] Master 127.16.40.252:37141 requested a full tablet report, sending...
I20260812 18:54:08.487546 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.466627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51335 (local address 127.16.40.254:41697)
0812 18:54:08.466765 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.466770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.466916 (+ 146us) server_negotiation.cc:407] Connection header received
0812 18:54:08.467755 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.467758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.467814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.467900 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.469273 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.470087 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.478642 (+ 8555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.479028 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.479126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.481850 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.481854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.481857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.481888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.483645 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.484361 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.484365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.484366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.484455 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.484836 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.484842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.484847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.485120 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.485212 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.486979 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.487133 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:08.489019 24705 ts_manager.cc:194] Registered new tserver with Master: e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193:43461)
I20260812 18:54:08.489058 24490 ts_manager.cc:194] Registered new tserver with Master: e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193:43461)
I20260812 18:54:08.489957 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.466525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:51335)
0812 18:54:08.466826 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.466839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.467162 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.467957 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.467961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.468407 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.469106 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.469111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.470199 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.470203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.478449 (+ 8246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.478465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.479360 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.479366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.479483 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.480015 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.480031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.481668 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.483803 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.483805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.483807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.484235 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.484589 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.484596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.484600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.484723 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.486556 (+ 1833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.486559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.486727 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.487805 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.489754 (+ 1949us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":6886,"spinlock_wait_cycles":101632,"thread_start_us":115,"threads_started":1}
I20260812 18:54:08.490670 24915 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:08.490775 24915 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.491015 24915 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
I20260812 18:54:08.491905 23775 ts_manager.cc:194] Re-registered known tserver with Master: e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193:43461)
I20260812 18:54:08.492535 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:51335
I20260812 18:54:08.493928 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:39609
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:41697
--webserver_port=39793
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:54:08.577688 24311 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
W20260812 18:54:08.620357 24924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:08.620738 24924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:08.620790 24924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:08.625049 24924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:08.625128 24924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:08.625191 24924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:08.630904 24924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:39609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=39793
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:08.632109 24924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:08.633180 24924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:08.644038 24933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.644098 24930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.644086 24931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.644433 24924 server_base.cc:1034] running on GCE node
I20260812 18:54:08.644932 24924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:08.645689 24924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:08.646895 24924 hybrid_clock.cc:648] HybridClock initialized: now 1786560848646856 us; error 53 us; skew 500 ppm
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:08.650076 24924 init.cc:377] Logged in from keytab as oryx/127.16.40.194@KRBTEST.COM (short username oryx)
I20260812 18:54:08.651070 24924 webserver.cc:460] Webserver started at http://127.16.40.194:39793/ using document root <none> and password file <none>
I20260812 18:54:08.651647 24924 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.651727 24924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.657521 24924 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:08.661252 24941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.662907 24924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:54:08.663102 24924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "cb40b77756fa4e42a033bdca76a48db5"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "b90cd74942b9a8d41f902b0030977cb6"
server_key_iv: "9041a08ee09a35ec903b42da3100b4a4"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.663751 24924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.697026 24924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.700847 24924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:08.701107 24924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.701781 24924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:08.702749 24924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:08.702821 24924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.702891 24924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:08.702934 24924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.712637 24924 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:39609
I20260812 18:54:08.712663 25078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:39609 every 8 connection(s)
I20260812 18:54:08.713847 24924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:08.722839 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 24924
I20260812 18:54:08.723070 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24162
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:08.733017 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.715911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:55527 (local address 127.16.40.254:41697)
0812 18:54:08.716088 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.716093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.717189 (+ 1096us) server_negotiation.cc:407] Connection header received
0812 18:54:08.718382 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.718386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.718464 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.718564 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.719965 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.720833 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.722645 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.723046 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.723140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.727656 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.727660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.727663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.727702 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.729737 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.730700 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.730705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.730707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.730854 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.731550 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.731554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.731556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.731799 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.731878 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.732693 (+ 815us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.732836 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
I20260812 18:54:08.734789 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.716395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:55527)
0812 18:54:08.717004 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.717066 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.718165 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.718745 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.718752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.719258 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.719810 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.719824 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.721229 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.721233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.722472 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.722484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.723432 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.723438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.725043 (+ 1605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.725761 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.725780 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.727492 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.729896 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.729904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.729926 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.730504 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.731043 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.731047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.731049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.731364 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.731970 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.731978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.732339 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.733214 (+ 875us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.733630 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1574,"spinlock_wait_cycles":29952,"thread_start_us":140,"threads_started":1}
I20260812 18:54:08.738343 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.715860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:38069 (local address 127.16.40.253:41923)
0812 18:54:08.716017 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.716023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.717192 (+ 1169us) server_negotiation.cc:407] Connection header received
0812 18:54:08.718404 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.718408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.718464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.718535 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.720009 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.720661 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.722645 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.723056 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.723138 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.727339 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.727349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.727355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.727414 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.736299 (+ 8885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.736899 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.736904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.736906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.737008 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.737353 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.737356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.737358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.737619 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.737704 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.738038 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.738173 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 18:54:08.738960 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.716395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.16.40.194:38069)
0812 18:54:08.717004 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.717066 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.718180 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.718746 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.718755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.719296 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.719829 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.719840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.720812 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.720816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.722472 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.722484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.723432 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.723438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.723708 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.724581 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.724608 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.727043 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.736444 (+ 9401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.736447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.736450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.736765 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.737145 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.737148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.737150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.737243 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.737763 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.737766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.737864 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.738565 (+ 701us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.738731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":257,"spinlock_wait_cycles":22528,"thread_start_us":202,"threads_started":1}
I20260812 18:54:08.740306 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.717792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:60181 (local address 127.16.40.252:37141)
0812 18:54:08.717931 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.717938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.718044 (+ 106us) server_negotiation.cc:407] Connection header received
0812 18:54:08.718401 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.718405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.718465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.718525 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.720016 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.720830 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.722326 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.722648 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.722761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.734327 (+ 11566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.734335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.734341 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.734396 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.737235 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.737978 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.737985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.737990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.738106 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.738806 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.738811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.738815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.739058 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.739136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.739915 (+ 779us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.740073 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:08.740310 25081 heartbeater.cc:347] Connected to a master server at 127.16.40.253:41923
I20260812 18:54:08.740605 25081 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.741227 25081 heartbeater.cc:511] Master 127.16.40.253:41923 requested a full tablet report, sending...
I20260812 18:54:08.741622 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.717630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.16.40.194:60181)
0812 18:54:08.717929 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.717947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.718180 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.718746 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.718755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.719296 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.719830 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.719840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.720960 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.720963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.722157 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.722168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.730964 (+ 8796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.730970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.731128 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.731714 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.731737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.734089 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.737449 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.737456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.737461 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.737818 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.738259 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.738265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.738270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.738383 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.739595 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.739601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.739718 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.740777 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.740960 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":186,"spinlock_wait_cycles":10112,"thread_start_us":96,"threads_started":1}
I20260812 18:54:08.742671 24490 ts_manager.cc:194] Registered new tserver with Master: cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:08.743139 25079 heartbeater.cc:347] Connected to a master server at 127.16.40.252:37141
I20260812 18:54:08.743239 25079 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.743434 25079 heartbeater.cc:511] Master 127.16.40.252:37141 requested a full tablet report, sending...
I20260812 18:54:08.744258 24705 ts_manager.cc:194] Registered new tserver with Master: cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:08.745255 25080 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:08.745342 25080 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.745618 25080 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
I20260812 18:54:08.746331 23775 ts_manager.cc:194] Re-registered known tserver with Master: cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:08.747006 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:55527
I20260812 18:54:08.747840 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:39195
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:41697
--webserver_port=36951
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:54:08.826705 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.822814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.16.40.252:37141)
0812 18:54:08.822985 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.822991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.823093 (+ 102us) server_negotiation.cc:407] Connection header received
0812 18:54:08.823301 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.823305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.823367 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.823471 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:08.823779 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.824426 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.825647 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.826123 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.826244 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.826419 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.826489 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.826560 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 18:54:08.826709 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.822739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:42806)
0812 18:54:08.822985 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.823002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.823176 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.823486 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.823489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.823512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:08.823648 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.823670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.824578 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.824584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.825516 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.825527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.826256 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.826260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.826350 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.826374 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.826473 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.826560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260812 18:54:08.827715 24718 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 18:54:08.828117 24752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869ed47ea6194f9194abb85fcd53ea45. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.828286 24752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: LEARNER
W20260812 18:54:08.869321 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:08.869647 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:08.869745 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:08.874198 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:08.874264 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:08.874326 25087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:08.879814 25087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:39195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=36951
--tserver_master_addrs=127.16.40.254:41697,127.16.40.253:41923,127.16.40.252:37141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:08.881035 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:08.882081 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:08.892843 25095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.893038 25087 server_base.cc:1034] running on GCE node
W20260812 18:54:08.892843 25101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:08.892871 25094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:08.893568 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:08.894318 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:08.895541 25087 hybrid_clock.cc:648] HybridClock initialized: now 1786560848895495 us; error 60 us; skew 500 ppm
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:08.898612 25087 init.cc:377] Logged in from keytab as oryx/127.16.40.195@KRBTEST.COM (short username oryx)
I20260812 18:54:08.899595 25087 webserver.cc:460] Webserver started at http://127.16.40.195:36951/ using document root <none> and password file <none>
I20260812 18:54:08.900203 25087 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:08.900315 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:08.904855 25087 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:08.907325 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.908923 25087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:08.909076 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "7a176f8855fb4addba2354a26034574a"
format_stamp: "Formatted at 2026-08-12 18:54:07 on dist-test-slave-rnrw"
server_key: "12009c99639d98456bdc192f695f6c11"
server_key_iv: "fac2df9fa6276781feda4b261aa0bde9"
server_key_version: "encryptionkey@0"
I20260812 18:54:08.909700 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:08.921133 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:08.924489 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:08.924741 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:08.925443 25087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:08.926350 25087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:08.926419 25087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.926492 25087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:08.926532 25087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:08.936221 25087 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39195
I20260812 18:54:08.936241 25236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39195 every 8 connection(s)
I20260812 18:54:08.937390 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:08.945338 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 25087
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.253@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
Aug 12 18:54:08 dist-test-slave-rnrw krb5kdc[23727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560848, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.252@KRBTEST.COM
I20260812 18:54:08.956794 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.940853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:37305 (local address 127.16.40.252:37141)
0812 18:54:08.941035 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.941042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.941070 (+ 28us) server_negotiation.cc:407] Connection header received
0812 18:54:08.941505 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.941511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.941570 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.941643 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.943127 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.943917 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945200 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945544 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945626 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.951889 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.951893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.951896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.951922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.953641 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.954725 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.954729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.954731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.954818 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955314 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.955320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.955324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.955566 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.955646 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.956450 (+ 804us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.956625 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 18:54:08.958052 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.940727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.16.40.195:37305)
0812 18:54:08.940984 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.940997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.941198 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.941745 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.941753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.942320 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.942948 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.942957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.944077 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.944080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.944990 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.945000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945860 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.946110 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.947006 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.947027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.951727 (+ 4700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.953764 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.953774 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.953792 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.954619 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.954908 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.954913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.954915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.955158 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.955728 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.955736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.956152 (+ 416us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.956999 (+ 847us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.957136 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":288,"spinlock_wait_cycles":61312,"thread_start_us":98,"threads_started":1}
I20260812 18:54:08.958338 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.939650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:51579)
0812 18:54:08.940143 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.940198 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.941197 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.941930 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.941933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.942314 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.942937 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.942952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.944035 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.944038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.945100 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.945109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945822 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.946151 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.947061 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.947081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.950309 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.954932 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.954934 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.954936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.955178 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.955685 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.955689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.955773 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.956243 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.956246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.956344 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.956494 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.956890 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":301,"spinlock_wait_cycles":11264,"thread_start_us":142,"threads_started":1}
I20260812 18:54:08.959568 25237 heartbeater.cc:347] Connected to a master server at 127.16.40.252:37141
I20260812 18:54:08.959868 25237 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.959978 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.939217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:51579 (local address 127.16.40.254:41697)
0812 18:54:08.939373 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.939379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.940305 (+ 926us) server_negotiation.cc:407] Connection header received
0812 18:54:08.941765 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.941767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.941810 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.941898 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.943096 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.943917 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945225 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945570 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945663 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.950437 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.950442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.950446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.950479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.954817 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955494 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.955497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.955499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.955580 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955873 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.955877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.955878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.956103 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.956199 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.959674 (+ 3475us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.959804 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:08.960305 25249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.947568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.16.40.254:41697)
0812 18:54:08.948890 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.948895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.949070 (+ 175us) server_negotiation.cc:407] Connection header received
0812 18:54:08.949119 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.949125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.949180 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.949269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.950425 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.951288 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.952665 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.953039 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.953149 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.955796 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.955800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.955802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.955832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.957678 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.958671 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.958675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.958677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.958766 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.959088 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.959095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.959097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.959308 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.959434 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.960027 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.960148 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":91,"threads_started":1}
I20260812 18:54:08.960577 25237 heartbeater.cc:511] Master 127.16.40.252:37141 requested a full tablet report, sending...
I20260812 18:54:08.961190 25238 heartbeater.cc:347] Connected to a master server at 127.16.40.254:41697
I20260812 18:54:08.961288 25238 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.961488 25238 heartbeater.cc:511] Master 127.16.40.254:41697 requested a full tablet report, sending...
I20260812 18:54:08.961532 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.939209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:52115 (local address 127.16.40.253:41923)
0812 18:54:08.939344 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.939349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.940307 (+ 958us) server_negotiation.cc:407] Connection header received
0812 18:54:08.941387 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.941391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.941454 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.941550 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.943114 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.943649 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945221 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945580 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945654 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.950152 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.950158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.950161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.950196 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.953502 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955040 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.955043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.955046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.955143 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.958731 (+ 3588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.958735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.958737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.958979 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.959077 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.961270 (+ 2193us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.961405 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:54:08.963325 23775 ts_manager.cc:194] Re-registered known tserver with Master: 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:08.963440 24705 ts_manager.cc:194] Registered new tserver with Master: 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:08.963920 23775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:51579
I20260812 18:54:08.964397 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.939650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.16.40.195:52115)
0812 18:54:08.940143 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.940198 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.941189 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.941745 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.941753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.942317 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:08.942950 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.942957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.943800 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.943802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.945075 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.945086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.945878 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.945883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.946128 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.947168 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:08.947186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:08.949955 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:08.953849 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.953852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.953854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.954227 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.955280 (+ 1053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:08.955285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:08.955290 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:08.955381 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:08.959125 (+ 3744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:08.959130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:08.959224 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.959338 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.959487 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":218,"spinlock_wait_cycles":129280,"thread_start_us":147,"threads_started":1}
I20260812 18:54:08.966325 25239 heartbeater.cc:347] Connected to a master server at 127.16.40.253:41923
I20260812 18:54:08.966423 25239 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:08.966605 25239 heartbeater.cc:511] Master 127.16.40.253:41923 requested a full tablet report, sending...
I20260812 18:54:08.967366 24490 ts_manager.cc:194] Registered new tserver with Master: 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:08.963968 23774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 18:54:08.968308 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.948151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.16.40.253:41923)
0812 18:54:08.948419 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.948426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.948442 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:08.948479 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.948482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.948528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.949664 (+ 1136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.949764 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.955802 (+ 6038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.957191 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.957599 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.958096 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.962721 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.962725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.962727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.962752 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.965707 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.966151 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.966154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.966156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.966241 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.967101 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.967104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.967106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.967311 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.967492 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.968003 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.968141 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260812 18:54:08.968737 23774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:08.970613 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.948263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.16.40.252:37141)
0812 18:54:08.948602 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.948607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.949132 (+ 525us) server_negotiation.cc:407] Connection header received
0812 18:54:08.949325 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.949328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.949379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.949666 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:08.950966 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.951820 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.953210 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.953584 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.953667 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.959029 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:08.959032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:08.959036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:08.959077 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:08.960990 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.961847 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.961853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.961854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.961943 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:08.962254 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:08.962259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:08.962263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:08.962963 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:08.963099 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.970361 (+ 7262us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.970488 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260812 18:54:08.971426 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: NON_VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: true } } }
I20260812 18:54:08.971581 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.975687 25088 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: attempting to promote NON_VOTER 7844627255b44fa98e1f07504f7c87a8 to VOTER
I20260812 18:54:08.976044 25088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:08.977211 24496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 18:54:08.977608 25256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 18:54:08.977739 24718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 LEARNER]: Refusing update from remote peer 869ed47ea6194f9194abb85fcd53ea45: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 18:54:08.978085 25258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 18:54:08.979244 25266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.979722 24496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.980522 24718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.981410 25258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.981519 25258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.981789 25258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.981945 25258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.981945 25256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.982098 25256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:08.982367 24752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.982486 24752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:08.983004 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869ed47ea6194f9194abb85fcd53ea45" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:08.983896 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:08.990171 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.983888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43461 (local address 127.0.0.1:57140)
0812 18:54:08.984019 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.984036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.984149 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.986194 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.986197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.986220 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:08.986310 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.986317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.987663 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.987666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.988659 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.988670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.989787 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.989790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.989877 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.989899 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.989955 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.990034 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 18:54:08.990404 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.984212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39609 (local address 127.0.0.1:46636)
0812 18:54:08.985418 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.985432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.985535 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.985963 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.985965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.985985 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:08.986101 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.986111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.987515 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.987518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.988737 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.988747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.989980 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.989990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.990095 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.990118 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.990189 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.990261 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":94,"threads_started":1}
I20260812 18:54:08.990900 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.984678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.16.40.194:39609)
0812 18:54:08.985260 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.985272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.985517 (+ 245us) server_negotiation.cc:407] Connection header received
0812 18:54:08.985636 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.985641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.985827 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.985960 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:08.986226 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.987373 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.988840 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.989639 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.989763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.989920 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.990675 (+ 755us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.990767 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":98,"threads_started":1}
I20260812 18:54:08.990891 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.984335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.16.40.193:43461)
0812 18:54:08.985739 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.985753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.985775 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:54:08.985838 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.985842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.986028 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.986191 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:08.986430 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.987494 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.988835 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.989649 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.990441 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.990615 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.990672 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.990763 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1316,"thread_start_us":85,"threads_started":1}
I20260812 18:54:08.991204 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.984505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39195 (local address 127.0.0.1:35778)
0812 18:54:08.986426 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:08.986439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:08.986580 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:08.987199 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:08.987202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:08.987218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:08.987294 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.987299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.988874 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.988877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.989600 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:08.989609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.990764 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.990767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:08.990855 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.990885 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 18:54:08.990969 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:54:08.991040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1839,"thread_start_us":142,"threads_started":1}
I20260812 18:54:08.992691 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:08.985107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.16.40.195:39195)
0812 18:54:08.985455 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:08.985468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:08.986535 (+ 1067us) server_negotiation.cc:407] Connection header received
0812 18:54:08.986756 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:08.986761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:08.986936 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:08.987074 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:08.987435 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.988732 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.989725 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:08.990412 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:08.990533 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:08.990697 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:08.991468 (+ 771us) server_negotiation.cc:299] Negotiation successful
0812 18:54:08.991572 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":156,"threads_started":1}
I20260812 18:54:08.994068 24995 tablet_service.cc:1467] Processing CreateTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 (DEFAULT_TABLE table=test-table [id=2b96284ca8ac4e479bdc0cbb8e05047e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:08.994050 24840 tablet_service.cc:1467] Processing CreateTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 (DEFAULT_TABLE table=test-table [id=2b96284ca8ac4e479bdc0cbb8e05047e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:08.994792 25160 tablet_service.cc:1467] Processing CreateTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 (DEFAULT_TABLE table=test-table [id=2b96284ca8ac4e479bdc0cbb8e05047e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:08.995208 24840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5b8ea2d3864ead8f74ac753f784614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:08.995208 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5b8ea2d3864ead8f74ac753f784614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:08.995533 25160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5b8ea2d3864ead8f74ac753f784614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:09.002733 25275 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Bootstrap starting.
I20260812 18:54:09.003465 25277 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Bootstrap starting.
I20260812 18:54:09.004261 25276 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Bootstrap starting.
I20260812 18:54:09.005291 25275 tablet_bootstrap.cc:654] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:09.005667 25277 tablet_bootstrap.cc:654] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:09.006189 25276 tablet_bootstrap.cc:654] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:09.006481 25275 log.cc:824] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:09.006481 25277 log.cc:824] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:09.006983 25276 log.cc:824] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:09.008852 25277 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: No bootstrap required, opened a new log
I20260812 18:54:09.009019 25275 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: No bootstrap required, opened a new log
I20260812 18:54:09.009178 25277 ts_tablet_manager.cc:1397] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:54:09.009184 25276 tablet_bootstrap.cc:492] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: No bootstrap required, opened a new log
I20260812 18:54:09.009198 25275 ts_tablet_manager.cc:1397] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:54:09.009370 25276 ts_tablet_manager.cc:1397] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 18:54:09.013514 25277 raft_consensus.cc:348] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.014060 25277 raft_consensus.cc:374] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:09.014173 25277 raft_consensus.cc:729] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33f3265a9ee42abb43f61a8bfb5175a, State: Initialized, Role: FOLLOWER
I20260812 18:54:09.014155 25275 raft_consensus.cc:348] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.014163 25276 raft_consensus.cc:348] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.014420 25275 raft_consensus.cc:374] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:09.014425 25276 raft_consensus.cc:374] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:09.014473 25275 raft_consensus.cc:729] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a176f8855fb4addba2354a26034574a, State: Initialized, Role: FOLLOWER
I20260812 18:54:09.014483 25276 raft_consensus.cc:729] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb40b77756fa4e42a033bdca76a48db5, State: Initialized, Role: FOLLOWER
I20260812 18:54:09.014959 25277 consensus_queue.cc:260] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.014976 25276 consensus_queue.cc:260] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.014959 25275 consensus_queue.cc:260] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.016069 25276 ts_tablet_manager.cc:1428] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 18:54:09.016088 25275 ts_tablet_manager.cc:1428] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 18:54:09.016101 25277 ts_tablet_manager.cc:1428] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:54:09.017549 25080 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
I20260812 18:54:09.017897 24915 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
I20260812 18:54:09.022583 25238 heartbeater.cc:503] Master 127.16.40.254:41697 was elected leader, sending a full tablet report...
I20260812 18:54:09.090993 25284 raft_consensus.cc:482] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:09.091293 25284 raft_consensus.cc:504] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.092592 25284 leader_election.cc:288] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609), 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:09.098626 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.093078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:55613 (local address 127.16.40.194:39609)
0812 18:54:09.093219 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.093223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.093573 (+ 350us) server_negotiation.cc:407] Connection header received
0812 18:54:09.093652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.093656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.093731 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.093829 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:09.094349 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.095190 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.097246 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.097885 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.098064 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.098113 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.098384 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.098486 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 18:54:09.098666 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.093007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:60481 (local address 127.16.40.195:39195)
0812 18:54:09.093143 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.093150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.093574 (+ 424us) server_negotiation.cc:407] Connection header received
0812 18:54:09.093686 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.093689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.093739 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.093834 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:09.094349 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.095190 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.097238 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.097886 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.098045 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.098116 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.098386 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.098558 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 18:54:09.098677 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.092959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39609 (local address 127.16.40.193:55613)
0812 18:54:09.093358 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:09.093379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:09.093533 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:09.093895 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:09.093899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:09.093933 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:09.094195 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:09.094203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.095338 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.095343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:09.097065 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:09.097079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.098041 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.098044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:09.098179 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.098235 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 18:54:09.098322 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 18:54:09.098443 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":223,"spinlock_wait_cycles":14080,"thread_start_us":232,"threads_started":1}
I20260812 18:54:09.098677 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.092921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39195 (local address 127.16.40.193:60481)
0812 18:54:09.093358 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:09.093380 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:09.093533 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:09.093895 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:09.093899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:09.093933 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:09.094195 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:09.094203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.095338 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.095343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:09.097065 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:09.097090 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.098039 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.098043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:09.098183 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.098236 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 18:54:09.098324 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 18:54:09.098443 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":342,"spinlock_wait_cycles":24064,"thread_start_us":226,"threads_started":1}
I20260812 18:54:09.099715 25185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5b8ea2d3864ead8f74ac753f784614" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a176f8855fb4addba2354a26034574a" is_pre_election: true
I20260812 18:54:09.099720 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5b8ea2d3864ead8f74ac753f784614" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb40b77756fa4e42a033bdca76a48db5" is_pre_election: true
I20260812 18:54:09.100045 25185 raft_consensus.cc:2454] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 0.
I20260812 18:54:09.100056 25022 raft_consensus.cc:2454] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 0.
I20260812 18:54:09.100597 24788 leader_election.cc:302] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a176f8855fb4addba2354a26034574a, e33f3265a9ee42abb43f61a8bfb5175a; no voters:
I20260812 18:54:09.100914 25284 raft_consensus.cc:2781] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:09.101007 25284 raft_consensus.cc:482] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:09.101052 25284 raft_consensus.cc:3037] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:09.102185 25284 raft_consensus.cc:504] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.102550 25284 leader_election.cc:288] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 election: Requested vote from peers cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609), 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195)
I20260812 18:54:09.102947 25185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5b8ea2d3864ead8f74ac753f784614" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a176f8855fb4addba2354a26034574a"
I20260812 18:54:09.102981 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5b8ea2d3864ead8f74ac753f784614" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb40b77756fa4e42a033bdca76a48db5"
I20260812 18:54:09.103102 25185 raft_consensus.cc:3037] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:09.103145 25022 raft_consensus.cc:3037] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:09.104336 25022 raft_consensus.cc:2454] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 1.
I20260812 18:54:09.104339 25185 raft_consensus.cc:2454] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 1.
I20260812 18:54:09.104681 24788 leader_election.cc:302] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a176f8855fb4addba2354a26034574a, e33f3265a9ee42abb43f61a8bfb5175a; no voters:
I20260812 18:54:09.104878 25284 raft_consensus.cc:2781] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:09.105108 25284 raft_consensus.cc:686] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 1 LEADER]: Becoming Leader. State: Replica: e33f3265a9ee42abb43f61a8bfb5175a, State: Running, Role: LEADER
I20260812 18:54:09.105502 25284 consensus_queue.cc:237] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } }
I20260812 18:54:09.109565 23774 catalog_manager.cc:5495] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a reported cstate change: term changed from 0 to 1, leader changed from <none> to e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193). New cstate: current_term: 1 leader_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } health_report { overall_health: UNKNOWN } } }
I20260812 18:54:09.146142 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.142168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.16.40.193:43461)
0812 18:54:09.142306 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.142310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.142324 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:09.142406 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.142409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.142476 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.142575 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:09.142838 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.143399 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.144354 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.144667 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.144757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.145114 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:09.145698 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:09.145803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.145872 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.145935 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 18:54:09.153120 25022 raft_consensus.cc:1261] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Refusing update from remote peer e33f3265a9ee42abb43f61a8bfb5175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:09.153119 25185 raft_consensus.cc:1261] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Refusing update from remote peer e33f3265a9ee42abb43f61a8bfb5175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:09.154073 25284 consensus_queue.cc:1035] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:09.154342 25288 consensus_queue.cc:1035] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:09.165777 25293 mvcc.cc:205] Tried to move back new op lower bound from 7317753238122160128 to 7317753237939367936. Current Snapshot: MvccSnapshot[applied={T|T < 7317753238122160128}]
I20260812 18:54:09.175583 23775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:09.175755 23775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:09.182021 23775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869ed47ea6194f9194abb85fcd53ea45: Sending DeleteTablet for 3 replicas of tablet ee5b8ea2d3864ead8f74ac753f784614
I20260812 18:54:09.182917 25160 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:09 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:35778
I20260812 18:54:09.182881 24840 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:09 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57140
I20260812 18:54:09.183058 24995 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5b8ea2d3864ead8f74ac753f784614 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:09 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:46636
I20260812 18:54:09.183604 25303 tablet_replica.cc:330] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: stopping tablet replica
I20260812 18:54:09.183941 25303 raft_consensus.cc:2229] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:09.184288 25303 raft_consensus.cc:2258] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:09.186131 25303 ts_tablet_manager.cc:1905] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:09.186334 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 23741
I20260812 18:54:09.191373 25302 tablet_replica.cc:330] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: stopping tablet replica
I20260812 18:54:09.191776 25302 raft_consensus.cc:2229] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:09.192171 25302 raft_consensus.cc:2258] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:09.192266 25303 ts_tablet_manager.cc:1918] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:09.192389 25303 log.cc:1196] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/ee5b8ea2d3864ead8f74ac753f784614
I20260812 18:54:09.192799 25301 tablet_replica.cc:330] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: stopping tablet replica
I20260812 18:54:09.192822 25303 ts_tablet_manager.cc:1939] T ee5b8ea2d3864ead8f74ac753f784614 P cb40b77756fa4e42a033bdca76a48db5: Deleting consensus metadata
I20260812 18:54:09.193114 25301 raft_consensus.cc:2229] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:09.193496 25301 raft_consensus.cc:2258] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:09.194911 25301 ts_tablet_manager.cc:1905] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:09.197827 25301 ts_tablet_manager.cc:1918] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:09.197944 25301 log.cc:1196] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/ee5b8ea2d3864ead8f74ac753f784614
I20260812 18:54:09.198292 25301 ts_tablet_manager.cc:1939] T ee5b8ea2d3864ead8f74ac753f784614 P e33f3265a9ee42abb43f61a8bfb5175a: Deleting consensus metadata
I20260812 18:54:09.199460 25302 ts_tablet_manager.cc:1905] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:09.202867 25302 ts_tablet_manager.cc:1918] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260812 18:54:09.202991 24943 connection.cc:535] server connection from 127.0.0.1:46636 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 18:54:09.203135 25302 log.cc:1196] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/ee5b8ea2d3864ead8f74ac753f784614
W20260812 18:54:09.203444 24787 connection.cc:535] server connection from 127.0.0.1:57140 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 18:54:09.203665 25302 ts_tablet_manager.cc:1939] T ee5b8ea2d3864ead8f74ac753f784614 P 7a176f8855fb4addba2354a26034574a: Deleting consensus metadata
W20260812 18:54:09.205194 25112 connection.cc:441] server connection from 127.0.0.1:35778 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35778 (request call id 1) could send its response
I20260812 18:54:09.206897 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.185246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.16.40.253:41923)
0812 18:54:09.185369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.185374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.196974 (+ 11600us) server_negotiation.cc:407] Connection header received
0812 18:54:09.197053 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.197056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.197124 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.197253 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:09.198317 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.199110 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.200294 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.200702 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.200828 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.202927 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:09.202931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:09.202934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:09.202969 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:09.204849 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.205426 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.205430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.205432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.205526 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.205841 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.205844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.205846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.206121 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:09.206256 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.206574 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.206738 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:54:09.218503 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.185654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.16.40.252:37141)
0812 18:54:09.185766 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.185770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.206899 (+ 21129us) server_negotiation.cc:407] Connection header received
0812 18:54:09.207119 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.207122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.207190 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.207284 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:09.208092 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.208939 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.210441 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.210856 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.210972 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.212856 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:09.212862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:09.212864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:09.212892 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:09.215234 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.215790 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.215794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.215795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.216056 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.216909 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.216915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.216917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.217192 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:09.217337 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.218101 (+ 764us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.218273 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:54:09.219619 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.209043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.16.40.252:37141)
0812 18:54:09.210148 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.210154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.210166 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:54:09.210207 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.210213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.210266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.210339 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:09.211130 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.211905 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.213584 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.213929 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.214004 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.215877 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:09.215882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:09.215884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:09.215913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:09.217697 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.218219 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.218222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.218224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.218310 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.218612 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.218616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.218617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.218844 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:09.218954 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.219340 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.219461 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002}
I20260812 18:54:09.221630 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:09.209383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.16.40.253:41923)
0812 18:54:09.209495 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:09.209500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:09.210167 (+ 667us) server_negotiation.cc:407] Connection header received
0812 18:54:09.210253 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:09.210256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:09.210304 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:09.210386 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:09.213179 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.213993 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.215340 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:09.215702 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:09.215783 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:09.217722 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:09.217726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:09.217730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:09.217766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:09.219622 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.220306 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.220310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.220312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.220405 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:09.220728 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:09.220732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:09.220734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:09.220924 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:09.221069 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:09.221311 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 18:54:09.221448 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:54:10.172287 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.171563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:56637)
0812 18:54:10.172023 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.172136 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":82,"threads_started":1}
W20260812 18:54:10.172983 25080 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:41697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.173319 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.172308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:44225)
0812 18:54:10.173061 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.173204 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":68,"threads_started":1}
I20260812 18:54:10.173434 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.172742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:60827)
0812 18:54:10.173228 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.173317 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":89,"threads_started":1}
W20260812 18:54:10.173763 25238 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:41697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
W20260812 18:54:10.174032 24915 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:41697 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.681922 25318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869ed47ea6194f9194abb85fcd53ea45)
I20260812 18:54:10.681950 25319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 869ed47ea6194f9194abb85fcd53ea45)
I20260812 18:54:10.682188 25319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:10.682188 25318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:10.683238 25318 leader_election.cc:288] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252:37141)
I20260812 18:54:10.683243 25319 leader_election.cc:288] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253:41923)
I20260812 18:54:10.684188 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.683744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58460)
0812 18:54:10.683980 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.684077 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260812 18:54:10.684269 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.683389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58464)
0812 18:54:10.684064 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.684146 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":81,"threads_started":1}
W20260812 18:54:10.684377 24691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
W20260812 18:54:10.684449 24474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.685438 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.685201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58474)
0812 18:54:10.685315 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.685369 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 18:54:10.685559 24474 leader_election.cc:334] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.685879 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.685593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58476)
0812 18:54:10.685766 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.685816 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260812 18:54:10.685972 24691 leader_election.cc:334] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.687911 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.683480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:41923 (local address 127.0.0.1:55818)
0812 18:54:10.683805 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.683823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:10.683991 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:10.684403 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:10.684406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:10.684425 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:10.684531 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:10.684537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.685579 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.685583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:10.686599 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:10.686610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.687502 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.687506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:10.687606 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:10.687632 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:54:10.687716 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:54:10.687789 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":72,"threads_started":1}
I20260812 18:54:10.687966 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.683532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.16.40.253:41923)
0812 18:54:10.683806 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:10.683813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:10.683902 (+ 89us) server_negotiation.cc:407] Connection header received
0812 18:54:10.684127 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:10.684131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:10.684179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:10.684243 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:10.684708 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.685417 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.686730 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.687367 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.687502 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:10.687565 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:10.687734 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 18:54:10.687806 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":196,"thread_start_us":56,"threads_started":1}
I20260812 18:54:10.689117 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.683772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:37141 (local address 127.0.0.1:42836)
0812 18:54:10.684772 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.684788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:10.684941 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:10.685366 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:10.685369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:10.685389 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:10.685728 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:10.685741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.686692 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.686696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:10.687759 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:10.687769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.688740 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.688746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:10.688851 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:10.688883 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 18:54:10.688961 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 18:54:10.689013 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":67,"thread_start_us":57,"threads_started":1}
I20260812 18:54:10.689078 24496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7844627255b44fa98e1f07504f7c87a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" is_pre_election: true
I20260812 18:54:10.689206 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.684084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.16.40.252:37141)
0812 18:54:10.684516 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:10.684524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:10.684864 (+ 340us) server_negotiation.cc:407] Connection header received
0812 18:54:10.685084 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:10.685090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:10.685147 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:10.685211 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:10.685869 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.686567 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.687951 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:10.688599 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:10.688717 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:10.688775 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:10.689017 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 18:54:10.689097 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260812 18:54:10.689302 24496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7844627255b44fa98e1f07504f7c87a8 in term 1.
I20260812 18:54:10.689687 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7844627255b44fa98e1f07504f7c87a8" is_pre_election: true
I20260812 18:54:10.689689 24689 leader_election.cc:302] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be9bed7261d4c98babdceb0b6b3cae5, 7844627255b44fa98e1f07504f7c87a8; no voters: 869ed47ea6194f9194abb85fcd53ea45
I20260812 18:54:10.689893 24718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be9bed7261d4c98babdceb0b6b3cae5 in term 1.
I20260812 18:54:10.689989 25319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 18:54:10.690052 25319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869ed47ea6194f9194abb85fcd53ea45)
I20260812 18:54:10.690099 25319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:54:10.690255 24474 leader_election.cc:302] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be9bed7261d4c98babdceb0b6b3cae5, 7844627255b44fa98e1f07504f7c87a8; no voters: 869ed47ea6194f9194abb85fcd53ea45
I20260812 18:54:10.690475 25318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 18:54:10.690538 25318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 869ed47ea6194f9194abb85fcd53ea45)
I20260812 18:54:10.690577 25318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:54:10.691524 25319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:10.691977 25319 leader_election.cc:288] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 election: Requested vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 2be9bed7261d4c98babdceb0b6b3cae5 (127.16.40.253:41923)
I20260812 18:54:10.691922 25318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:10.692332 25318 leader_election.cc:288] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 election: Requested vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252:37141)
I20260812 18:54:10.692299 24496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7844627255b44fa98e1f07504f7c87a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2be9bed7261d4c98babdceb0b6b3cae5"
I20260812 18:54:10.692361 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.692104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58492)
0812 18:54:10.692219 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.692273 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 18:54:10.692482 24496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7844627255b44fa98e1f07504f7c87a8 in current term 2: Already voted for candidate 2be9bed7261d4c98babdceb0b6b3cae5 in this term.
I20260812 18:54:10.692672 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.692430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58502)
0812 18:54:10.692527 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.692606 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 18:54:10.692790 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7844627255b44fa98e1f07504f7c87a8"
I20260812 18:54:10.692948 24718 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2be9bed7261d4c98babdceb0b6b3cae5 in current term 2: Already voted for candidate 7844627255b44fa98e1f07504f7c87a8 in this term.
I20260812 18:54:10.693373 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.693030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58512)
0812 18:54:10.693249 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.693301 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260812 18:54:10.693464 24691 leader_election.cc:334] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.693519 24691 leader_election.cc:302] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7844627255b44fa98e1f07504f7c87a8; no voters: 2be9bed7261d4c98babdceb0b6b3cae5, 869ed47ea6194f9194abb85fcd53ea45
I20260812 18:54:10.693641 25319 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 18:54:10.693732 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:10.693467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58524)
0812 18:54:10.693560 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:10.693628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 18:54:10.693853 24474 leader_election.cc:334] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:10.693936 24474 leader_election.cc:302] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be9bed7261d4c98babdceb0b6b3cae5; no voters: 7844627255b44fa98e1f07504f7c87a8, 869ed47ea6194f9194abb85fcd53ea45
I20260812 18:54:10.694077 25318 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 18:54:11.174263 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:11.173692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:43225)
0812 18:54:11.174052 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:11.174130 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260812 18:54:11.174814 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:11.174289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:35001)
0812 18:54:11.174641 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:11.174710 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":95,"threads_started":1}
I20260812 18:54:11.174984 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:11.174537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:58919)
0812 18:54:11.174814 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:11.174876 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260812 18:54:12.175849 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.175354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:52669)
0812 18:54:12.175648 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.175723 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260812 18:54:12.175848 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.175479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:58867)
0812 18:54:12.175689 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.175744 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
I20260812 18:54:12.176159 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.175734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:47645)
0812 18:54:12.175997 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.176055 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260812 18:54:12.521077 25341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:12.521236 25341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:12.521754 25341 leader_election.cc:288] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252:37141)
I20260812 18:54:12.522395 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.521936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58790)
0812 18:54:12.522197 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.522261 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":64,"threads_started":1}
I20260812 18:54:12.522728 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7844627255b44fa98e1f07504f7c87a8" is_pre_election: true
I20260812 18:54:12.522873 24718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be9bed7261d4c98babdceb0b6b3cae5 in term 2.
I20260812 18:54:12.523363 24474 leader_election.cc:302] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be9bed7261d4c98babdceb0b6b3cae5, 7844627255b44fa98e1f07504f7c87a8; no voters:
I20260812 18:54:12.523586 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 18:54:12.523655 25341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:12.523706 25341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 2 FOLLOWER]: Advancing to term 3
I20260812 18:54:12.524299 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.523941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58796)
0812 18:54:12.524162 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.524227 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260812 18:54:12.524443 24474 leader_election.cc:334] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:12.525185 25341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:12.525552 25341 leader_election.cc:288] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 election: Requested vote from peers 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697), 7844627255b44fa98e1f07504f7c87a8 (127.16.40.252:37141)
I20260812 18:54:12.525943 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.525667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58798)
0812 18:54:12.525792 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.525864 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 18:54:12.526134 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7844627255b44fa98e1f07504f7c87a8"
I20260812 18:54:12.526290 24718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 2 FOLLOWER]: Advancing to term 3
I20260812 18:54:12.526744 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.526549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58812)
0812 18:54:12.526648 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.526693 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 18:54:12.526870 24474 leader_election.cc:334] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
I20260812 18:54:12.527516 24718 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be9bed7261d4c98babdceb0b6b3cae5 in term 3.
I20260812 18:54:12.527854 24474 leader_election.cc:302] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be9bed7261d4c98babdceb0b6b3cae5, 7844627255b44fa98e1f07504f7c87a8; no voters: 869ed47ea6194f9194abb85fcd53ea45
I20260812 18:54:12.528077 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 18:54:12.528288 25341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [term 3 LEADER]: Becoming Leader. State: Replica: 2be9bed7261d4c98babdceb0b6b3cae5, State: Running, Role: LEADER
I20260812 18:54:12.528685 25341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } }
I20260812 18:54:12.529762 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be9bed7261d4c98babdceb0b6b3cae5. Latest consensus state: current_term: 3 leader_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:12.529903 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:12.530228 25346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:12.532902 25346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:12.533460 25346 catalog_manager.cc:1259] Loaded cluster ID: fadcf02855ba43fdb8c82bfb05a0ea83
I20260812 18:54:12.533515 25346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:12.534282 25346 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:12.534703 25346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Loaded TSK: 0
I20260812 18:54:12.535486 25346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:54:12.619552 24488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:12.622848 24488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:12.626276 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.625986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58840)
0812 18:54:12.626138 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.626205 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 18:54:12.626400 24718 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [term 3 FOLLOWER]: Refusing update from remote peer 2be9bed7261d4c98babdceb0b6b3cae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 18:54:12.626995 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.626764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.0.0.1:58846)
0812 18:54:12.626868 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.626929 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 18:54:12.627063 25344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 18:54:12.627373 24474 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 -> Peer 869ed47ea6194f9194abb85fcd53ea45 (127.16.40.254:41697): Couldn't send request to peer 869ed47ea6194f9194abb85fcd53ea45. Status: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 18:54:12.628666 25348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be9bed7261d4c98babdceb0b6b3cae5. Latest consensus state: current_term: 3 leader_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:12.628805 25348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:12.630061 25341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:12.630225 25341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:12.630438 25348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "869ed47ea6194f9194abb85fcd53ea45" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41697 } } peers { permanent_uuid: "2be9bed7261d4c98babdceb0b6b3cae5" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 41923 } attrs { promote: false } } peers { permanent_uuid: "7844627255b44fa98e1f07504f7c87a8" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 37141 } attrs { promote: false } } }
I20260812 18:54:12.630558 25348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7844627255b44fa98e1f07504f7c87a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:54:12.641366 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.636187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39195 (local address 127.0.0.1:35794)
0812 18:54:12.636360 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.636375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:12.636482 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:12.637311 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:12.637317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:12.637341 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:12.637460 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.637471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.638613 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.638616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.639978 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.639988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.640963 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.640967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.641050 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.641079 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 18:54:12.641131 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 18:54:12.641215 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":109}
I20260812 18:54:12.641726 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.636580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43461 (local address 127.0.0.1:57160)
0812 18:54:12.636825 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.636841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:12.636935 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:12.637445 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:12.637447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:12.637462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:12.637553 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.637559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.638537 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.638540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.639969 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.639979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.641170 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.641275 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.641303 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 18:54:12.641391 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 18:54:12.641593 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":139,"thread_start_us":113,"threads_started":1}
I20260812 18:54:12.641942 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.636329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.16.40.195:39195)
0812 18:54:12.637042 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:12.637050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:12.637071 (+ 21us) server_negotiation.cc:407] Connection header received
0812 18:54:12.637123 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:12.637126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:12.637176 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:12.637582 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:12.637702 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.638523 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.640126 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.640819 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641581 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.641663 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:12.641704 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 18:54:12.641784 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":116,"threads_started":1}
I20260812 18:54:12.642110 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.636944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.16.40.193:43461)
0812 18:54:12.637187 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:12.637192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:12.637205 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:12.637265 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:12.637271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:12.637322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:12.637438 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:12.637675 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.638436 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.640270 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.641041 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.641246 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:12.641410 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 18:54:12.641950 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260812 18:54:12.642365 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.636863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39609 (local address 127.0.0.1:46646)
0812 18:54:12.637216 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.637231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:12.637311 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:12.638190 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:12.638194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:12.638213 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:12.638319 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.638324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.639733 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.639736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.640886 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:12.640899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.641956 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:12.642076 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.642102 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:54:12.642176 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 18:54:12.642251 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":93,"threads_started":1}
I20260812 18:54:12.642468 25160 tablet_service.cc:1467] Processing CreateTablet for tablet f113dad235544a43a6b7fa9b948908eb (DEFAULT_TABLE table=test-table [id=08084b40d24147a892e1803734809cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:12.642607 24840 tablet_service.cc:1467] Processing CreateTablet for tablet f113dad235544a43a6b7fa9b948908eb (DEFAULT_TABLE table=test-table [id=08084b40d24147a892e1803734809cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:12.642802 25160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f113dad235544a43a6b7fa9b948908eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:12.642838 24840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f113dad235544a43a6b7fa9b948908eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:12.642871 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.637218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.16.40.194:39609)
0812 18:54:12.637956 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:12.637961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:12.637975 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:12.638025 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:12.638027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:12.638075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:12.638165 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:12.638440 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.639021 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641023 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.641661 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.641762 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.642608 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:12.642650 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 18:54:12.642741 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":80,"threads_started":1}
I20260812 18:54:12.643416 24995 tablet_service.cc:1467] Processing CreateTablet for tablet f113dad235544a43a6b7fa9b948908eb (DEFAULT_TABLE table=test-table [id=08084b40d24147a892e1803734809cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:12.643654 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f113dad235544a43a6b7fa9b948908eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:12.645903 25363 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a: Bootstrap starting.
I20260812 18:54:12.646368 25365 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Bootstrap starting.
I20260812 18:54:12.646745 25364 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Bootstrap starting.
I20260812 18:54:12.646958 25363 tablet_bootstrap.cc:654] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:12.647324 25365 tablet_bootstrap.cc:654] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:12.647717 25364 tablet_bootstrap.cc:654] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:12.648375 25363 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a: No bootstrap required, opened a new log
I20260812 18:54:12.648480 25363 ts_tablet_manager.cc:1397] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:12.648645 25365 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: No bootstrap required, opened a new log
I20260812 18:54:12.648746 25365 ts_tablet_manager.cc:1397] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 18:54:12.648919 25363 raft_consensus.cc:348] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.649055 25363 raft_consensus.cc:374] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:12.649094 25363 raft_consensus.cc:729] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33f3265a9ee42abb43f61a8bfb5175a, State: Initialized, Role: FOLLOWER
I20260812 18:54:12.649257 25363 consensus_queue.cc:260] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.649370 25364 tablet_bootstrap.cc:492] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: No bootstrap required, opened a new log
I20260812 18:54:12.649467 25364 ts_tablet_manager.cc:1397] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:12.649441 25365 raft_consensus.cc:348] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.649566 25365 raft_consensus.cc:374] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:12.649612 25365 raft_consensus.cc:729] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb40b77756fa4e42a033bdca76a48db5, State: Initialized, Role: FOLLOWER
I20260812 18:54:12.649765 25365 consensus_queue.cc:260] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.649983 25364 raft_consensus.cc:348] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.650097 25364 raft_consensus.cc:374] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:12.650153 25364 raft_consensus.cc:729] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a176f8855fb4addba2354a26034574a, State: Initialized, Role: FOLLOWER
I20260812 18:54:12.650156 25363 ts_tablet_manager.cc:1428] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 18:54:12.650166 25365 ts_tablet_manager.cc:1428] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:12.650331 25364 consensus_queue.cc:260] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.650714 25364 ts_tablet_manager.cc:1428] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:12.651742 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.650929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:60851)
0812 18:54:12.651584 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.651656 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568}
I20260812 18:54:12.651911 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.651405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:54947)
0812 18:54:12.651788 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.651848 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260812 18:54:12.652009 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.651631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:37703)
0812 18:54:12.651883 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.651939 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260812 18:54:12.652711 25376 raft_consensus.cc:482] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:12.652804 25376 raft_consensus.cc:504] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.653162 25376 leader_election.cc:288] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195), cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:12.653692 25185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f113dad235544a43a6b7fa9b948908eb" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a176f8855fb4addba2354a26034574a" is_pre_election: true
I20260812 18:54:12.653823 25185 raft_consensus.cc:2454] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 0.
I20260812 18:54:12.654165 24788 leader_election.cc:302] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a176f8855fb4addba2354a26034574a, e33f3265a9ee42abb43f61a8bfb5175a; no voters:
I20260812 18:54:12.654345 25376 raft_consensus.cc:2781] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:12.654402 25376 raft_consensus.cc:482] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:12.654440 25376 raft_consensus.cc:3037] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:12.654785 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f113dad235544a43a6b7fa9b948908eb" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb40b77756fa4e42a033bdca76a48db5" is_pre_election: true
I20260812 18:54:12.654955 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.654479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:48793)
0812 18:54:12.654832 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.654886 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260812 18:54:12.654930 25022 raft_consensus.cc:2454] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 0.
I20260812 18:54:12.655901 24916 heartbeater.cc:503] Master 127.16.40.253:41923 was elected leader, sending a full tablet report...
I20260812 18:54:12.655927 25376 raft_consensus.cc:504] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.656323 25376 leader_election.cc:288] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 election: Requested vote from peers 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195), cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609)
I20260812 18:54:12.656558 25081 heartbeater.cc:503] Master 127.16.40.253:41923 was elected leader, sending a full tablet report...
I20260812 18:54:12.657157 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.656853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:33957)
0812 18:54:12.657016 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.657077 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 18:54:12.657559 25185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f113dad235544a43a6b7fa9b948908eb" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a176f8855fb4addba2354a26034574a"
I20260812 18:54:12.657699 25185 raft_consensus.cc:3037] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:12.657851 25022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f113dad235544a43a6b7fa9b948908eb" candidate_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb40b77756fa4e42a033bdca76a48db5"
I20260812 18:54:12.657987 25022 raft_consensus.cc:3037] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:12.658746 25185 raft_consensus.cc:2454] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 1.
I20260812 18:54:12.658910 25022 raft_consensus.cc:2454] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33f3265a9ee42abb43f61a8bfb5175a in term 1.
I20260812 18:54:12.659058 24788 leader_election.cc:302] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a176f8855fb4addba2354a26034574a, e33f3265a9ee42abb43f61a8bfb5175a; no voters:
I20260812 18:54:12.659258 25376 raft_consensus.cc:2781] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:12.659368 25376 raft_consensus.cc:686] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [term 1 LEADER]: Becoming Leader. State: Replica: e33f3265a9ee42abb43f61a8bfb5175a, State: Running, Role: LEADER
I20260812 18:54:12.659507 25376 consensus_queue.cc:237] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } }
I20260812 18:54:12.660080 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.659785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:47203)
0812 18:54:12.659949 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.660007 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 18:54:12.661312 24489 catalog_manager.cc:5495] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a reported cstate change: term changed from 0 to 1, leader changed from <none> to e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193). New cstate: current_term: 1 leader_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e33f3265a9ee42abb43f61a8bfb5175a" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 } health_report { overall_health: UNKNOWN } } }
I20260812 18:54:12.677057 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.673845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.16.40.193:43461)
0812 18:54:12.674018 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:12.674022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:12.674039 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:54:12.674153 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:12.674155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:12.674204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:12.674287 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:12.674537 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.675067 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.675938 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:12.676161 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:12.676267 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:12.676532 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:12.676662 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:12.676739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:12.676832 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 18:54:12.676893 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 18:54:12.679471 25022 raft_consensus.cc:1261] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Refusing update from remote peer e33f3265a9ee42abb43f61a8bfb5175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:12.679481 25185 raft_consensus.cc:1261] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Refusing update from remote peer e33f3265a9ee42abb43f61a8bfb5175a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:12.679998 25376 consensus_queue.cc:1035] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb40b77756fa4e42a033bdca76a48db5" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:12.680184 25371 consensus_queue.cc:1035] T f113dad235544a43a6b7fa9b948908eb P e33f3265a9ee42abb43f61a8bfb5175a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a176f8855fb4addba2354a26034574a" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:12.681828 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.681517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:45981)
0812 18:54:12.681683 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.681744 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 18:54:12.683048 25239 heartbeater.cc:503] Master 127.16.40.253:41923 was elected leader, sending a full tablet report...
I20260812 18:54:12.683845 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.683449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:59471)
0812 18:54:12.683700 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.683762 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260812 18:54:12.684412 25380 mvcc.cc:205] Tried to move back new op lower bound from 7317753252570497024 to 7317753252494835712. Current Snapshot: MvccSnapshot[applied={T|T < 7317753252570497024}]
I20260812 18:54:12.685676 25381 mvcc.cc:205] Tried to move back new op lower bound from 7317753252570497024 to 7317753252494835712. Current Snapshot: MvccSnapshot[applied={T|T < 7317753252570497024}]
I20260812 18:54:12.685868 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.685242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:55081)
0812 18:54:12.685726 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.685792 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
I20260812 18:54:12.687126 25340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.686891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.193:42783)
0812 18:54:12.687004 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.687060 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 18:54:12.687211 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.686498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.194:34167)
0812 18:54:12.687075 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.687134 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498}
I20260812 18:54:12.687760 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.687495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41697 (local address 127.16.40.195:56107)
0812 18:54:12.687613 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.687669 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:41697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 18:54:12.695325 24489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:12.695492 24489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:12.699656 24489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2be9bed7261d4c98babdceb0b6b3cae5: Sending DeleteTablet for 3 replicas of tablet f113dad235544a43a6b7fa9b948908eb
I20260812 18:54:12.700433 25160 tablet_service.cc:1514] Processing DeleteTablet for tablet f113dad235544a43a6b7fa9b948908eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:12 UTC) from {username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:35794
I20260812 18:54:12.700596 24840 tablet_service.cc:1514] Processing DeleteTablet for tablet f113dad235544a43a6b7fa9b948908eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:12 UTC) from {username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:57160
I20260812 18:54:12.700944 25389 tablet_replica.cc:330] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: stopping tablet replica
I20260812 18:54:12.701112 25389 raft_consensus.cc:2229] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:12.701263 25389 raft_consensus.cc:2258] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:12.701282 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24656
I20260812 18:54:12.701294 24995 tablet_service.cc:1514] Processing DeleteTablet for tablet f113dad235544a43a6b7fa9b948908eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:12 UTC) from {username='oryx', principal='oryx/127.16.40.253@KRBTEST.COM'} at 127.0.0.1:46646
I20260812 18:54:12.703450 25389 ts_tablet_manager.cc:1905] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:12.705801 25391 tablet_replica.cc:330] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: stopping tablet replica
I20260812 18:54:12.705875 25389 ts_tablet_manager.cc:1918] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:12.705942 25389 log.cc:1196] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/f113dad235544a43a6b7fa9b948908eb
I20260812 18:54:12.705973 25391 raft_consensus.cc:2229] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:12.706161 25391 raft_consensus.cc:2258] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:12.706244 25389 ts_tablet_manager.cc:1939] T f113dad235544a43a6b7fa9b948908eb P 7a176f8855fb4addba2354a26034574a: Deleting consensus metadata
I20260812 18:54:12.706938 25391 ts_tablet_manager.cc:1905] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:12.707122 24471 catalog_manager.cc:4841] TS 7a176f8855fb4addba2354a26034574a (127.16.40.195:39195): tablet f113dad235544a43a6b7fa9b948908eb (table test-table [id=08084b40d24147a892e1803734809cd5]) successfully deleted
I20260812 18:54:12.709148 25391 ts_tablet_manager.cc:1918] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:12.709242 25391 log.cc:1196] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/f113dad235544a43a6b7fa9b948908eb
I20260812 18:54:12.709569 25391 ts_tablet_manager.cc:1939] T f113dad235544a43a6b7fa9b948908eb P cb40b77756fa4e42a033bdca76a48db5: Deleting consensus metadata
I20260812 18:54:12.710424 24474 catalog_manager.cc:4841] TS cb40b77756fa4e42a033bdca76a48db5 (127.16.40.194:39609): tablet f113dad235544a43a6b7fa9b948908eb (table test-table [id=08084b40d24147a892e1803734809cd5]) successfully deleted
I20260812 18:54:12.720191 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24924
I20260812 18:54:12.720430 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:12.720125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43461 (local address 127.0.0.1:57176)
0812 18:54:12.720262 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:12.720329 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 18:54:12.723013 24469 catalog_manager.cc:4584] TS e33f3265a9ee42abb43f61a8bfb5175a (127.16.40.193:43461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f113dad235544a43a6b7fa9b948908eb: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43461: connect: Connection refused (error 111)
I20260812 18:54:12.731007 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 25087
I20260812 18:54:12.742782 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24320
I20260812 18:54:12.752991 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 24445
2026-08-12T18:54:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7496 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:12 dist-test-slave-rnrw krb5kdc[25395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:12 dist-test-slave-rnrw krb5kdc[25395](info): set up 2 sockets
Aug 12 18:54:12 dist-test-slave-rnrw krb5kdc[25395](info): commencing operation
krb5kdc: starting...
W20260812 18:54:13.914274 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab.
Entry for principal oryx/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab.
I20260812 18:54:14.054229 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37101
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab
--principal=oryx/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:14.174575 25409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:14.174891 25409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:14.174942 25409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:14.179332 25409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:14.179416 25409 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 18:54:14.179435 25409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:14.179452 25409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:14.179469 25409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:14.179509 25409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:14.185427 25409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:37101
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.254.keytab
--principal=oryx/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:14.186645 25409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:14.187741 25409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:14.196902 25414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:14.197000 25409 server_base.cc:1034] running on GCE node
W20260812 18:54:14.196894 25415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:14.197018 25417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:14.197785 25409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:14.199115 25409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:14.200376 25409 hybrid_clock.cc:648] HybridClock initialized: now 1786560854200357 us; error 37 us; skew 500 ppm
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:14.208957 25409 init.cc:377] Logged in from keytab as oryx/127.16.40.254@KRBTEST.COM (short username oryx)
I20260812 18:54:14.210045 25409 webserver.cc:460] Webserver started at http://127.16.40.254:41529/ using document root <none> and password file <none>
I20260812 18:54:14.210649 25409 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:14.210716 25409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:14.210898 25409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:14.213174 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "feb4ce57b74d42519ec42ddd01e86ab2"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "37756aba23df6430599540c232c37084"
server_key_iv: "841a903f406d2e75806c84f434036d87"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.213754 25409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "feb4ce57b74d42519ec42ddd01e86ab2"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "37756aba23df6430599540c232c37084"
server_key_iv: "841a903f406d2e75806c84f434036d87"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.217900 25409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 18:54:14.220687 25425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:14.222131 25409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:54:14.222280 25409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "feb4ce57b74d42519ec42ddd01e86ab2"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "37756aba23df6430599540c232c37084"
server_key_iv: "841a903f406d2e75806c84f434036d87"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.222405 25409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:14.250151 25409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:14.253895 25409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:14.254142 25409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:14.262841 25409 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:37101
I20260812 18:54:14.262974 25500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:37101 every 8 connection(s)
I20260812 18:54:14.264142 25409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:14.267355 25502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:14.272944 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 25409
I20260812 18:54:14.273110 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:14.273471 16547 external_mini_cluster.cc:1428] Setting key 1d5f409009f54e1a73bf6ae818e95aae
I20260812 18:54:14.273988 25502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Bootstrap starting.
I20260812 18:54:14.276762 25502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:14.277580 25502 log.cc:824] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:14.279637 25502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: No bootstrap required, opened a new log
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:14.283329 25502 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } }
I20260812 18:54:14.283603 25502 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:14.283681 25502 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb4ce57b74d42519ec42ddd01e86ab2, State: Initialized, Role: FOLLOWER
I20260812 18:54:14.284162 25502 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } }
I20260812 18:54:14.284364 25502 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:14.284446 25502 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:14.284607 25502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:14.285693 25502 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } }
I20260812 18:54:14.286023 25502 leader_election.cc:302] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb4ce57b74d42519ec42ddd01e86ab2; no voters:
I20260812 18:54:14.286417 25502 leader_election.cc:288] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:14.286514 25507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:14.286701 25507 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [term 1 LEADER]: Becoming Leader. State: Replica: feb4ce57b74d42519ec42ddd01e86ab2, State: Running, Role: LEADER
I20260812 18:54:14.287007 25507 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } }
I20260812 18:54:14.287837 25502 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:14.289520 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.275058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.16.40.254:37101)
0812 18:54:14.275598 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:14.275614 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:14.275652 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:54:14.276334 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:14.276352 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:14.276785 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:14.277172 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:14.278107 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.279130 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.280068 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.280659 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.281297 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.283825 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:14.283832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:14.283848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:14.283889 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:14.285959 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.286459 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.286465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.286474 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.286642 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.287014 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.287017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.287021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.287501 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:14.287708 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:14.288235 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 18:54:14.288604 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":172,"threads_started":1}
I20260812 18:54:14.290849 25508 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } } }
I20260812 18:54:14.290894 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb4ce57b74d42519ec42ddd01e86ab2. Latest consensus state: current_term: 1 leader_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb4ce57b74d42519ec42ddd01e86ab2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 37101 } } }
I20260812 18:54:14.291296 25508 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:14.291296 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2 [sys.catalog]: This master's current role is: LEADER
W20260812 18:54:14.293773 25525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:54:14.293881 25525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:54:14.293967 25526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:14.294817 25526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:14.301051 25526 catalog_manager.cc:1347] Generated new cluster ID: d24a187a309a48bf828e4d7ea64d2f2d
I20260812 18:54:14.301146 25526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:14.309625 25526 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:14.310645 25526 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:14.314850 25526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Generated new TSK 0
I20260812 18:54:14.315804 25526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
I20260812 18:54:14.356803 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37101
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:14.480131 25532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:14.480468 25532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:14.480516 25532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:14.485127 25532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:14.485206 25532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:14.485250 25532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:54:14.485303 25532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:14.491523 25532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:14.492837 25532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:14.494061 25532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:14.505367 25537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:14.505367 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:14.505710 25542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:14.505827 25532 server_base.cc:1034] running on GCE node
I20260812 18:54:14.506978 25532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:14.507767 25532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:14.508949 25532 hybrid_clock.cc:648] HybridClock initialized: now 1786560854508903 us; error 60 us; skew 500 ppm
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:14.512192 25532 init.cc:377] Logged in from keytab as oryx/127.16.40.193@KRBTEST.COM (short username oryx)
I20260812 18:54:14.513280 25532 webserver.cc:460] Webserver started at http://127.16.40.193:41199/ using document root <none> and password file <none>
I20260812 18:54:14.513871 25532 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:14.513935 25532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:14.514128 25532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:14.516429 25532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "e610a259a4d44bf7954e42e14f7510e3"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "e57b2eec0b9ccb594a57823eeb39b81d"
server_key_iv: "adf94caeac5bacf8d8d4f7c33a71b2a9"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.517099 25532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "e610a259a4d44bf7954e42e14f7510e3"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "e57b2eec0b9ccb594a57823eeb39b81d"
server_key_iv: "adf94caeac5bacf8d8d4f7c33a71b2a9"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.521467 25532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:14.524673 25549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:14.526868 25532 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:14.527084 25532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "e610a259a4d44bf7954e42e14f7510e3"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "e57b2eec0b9ccb594a57823eeb39b81d"
server_key_iv: "adf94caeac5bacf8d8d4f7c33a71b2a9"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.527287 25532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:14.551944 25532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:14.555958 25532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:14.556190 25532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:14.557977 25532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:14.558063 25532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:14.558146 25532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:14.558187 25532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:14.570492 25532 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:45489
I20260812 18:54:14.570655 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:45489 every 8 connection(s)
I20260812 18:54:14.571622 25532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:14.573871 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.559087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:39809 (local address 127.16.40.254:37101)
0812 18:54:14.559230 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:14.559234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:14.560476 (+ 1242us) server_negotiation.cc:407] Connection header received
0812 18:54:14.561811 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:14.561815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:14.561884 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:14.561984 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:14.563924 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.564779 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.566288 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.566542 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.566720 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.569811 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:14.569816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:14.569819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:14.569856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:14.571569 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.572151 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.572155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.572156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.572248 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.572761 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.572765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.572766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.573005 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:14.573098 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:14.573616 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 18:54:14.573739 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:54:14.574767 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.559702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.193:39809)
0812 18:54:14.560286 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:14.560338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:14.561551 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:14.562271 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:14.562283 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:14.562853 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:14.563699 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.563727 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.564936 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.564940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.566151 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.566166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.566780 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.566784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.566984 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.567666 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:14.567683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:14.569612 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:14.571726 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.571733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.571750 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.572035 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.572390 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.572396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.572399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.572645 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.573136 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:14.573141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:14.573390 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 18:54:14.573551 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 18:54:14.573835 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":160,"threads_started":1}
I20260812 18:54:14.576041 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 25532
I20260812 18:54:14.576181 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:14.576408 25691 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37101
I20260812 18:54:14.576536 16547 external_mini_cluster.cc:1428] Setting key cf5104c621b6e173607da814c1139237
I20260812 18:54:14.576762 25691 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:14.577452 25691 heartbeater.cc:511] Master 127.16.40.254:37101 requested a full tablet report, sending...
I20260812 18:54:14.579180 25442 ts_manager.cc:194] Registered new tserver with Master: e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489)
I20260812 18:54:14.580816 25442 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:39809
WARNING: no policy specified for oryx/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:54:14.590340 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.581785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:39555 (local address 127.16.40.254:37101)
0812 18:54:14.581923 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:14.581930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:14.582219 (+ 289us) server_negotiation.cc:407] Connection header received
0812 18:54:14.582334 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:14.582336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:14.582378 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:14.582472 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:14.583003 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.583576 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.584403 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.584712 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.584825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.586123 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:14.586130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:14.586135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:14.586173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:14.588031 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.588753 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.588759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.588767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.588879 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.589206 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.589210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.589212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.589488 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:14.589625 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:14.590097 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 18:54:14.590211 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:54:14.590354 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.581719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.193:39555)
0812 18:54:14.582115 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:14.582131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:14.582248 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:14.582503 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:14.582506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:14.582772 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:14.582878 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.582885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.583705 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.583709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.584279 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.584288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.584835 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.584839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.584933 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.585567 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:14.585580 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:14.585944 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:14.588190 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.588193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.588195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.588611 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.589004 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.589006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.589008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.589100 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.589612 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:14.589616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:14.589805 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 18:54:14.590001 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 18:54:14.590168 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
I20260812 18:54:14.612459 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37101
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:14.735711 25701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:14.736018 25701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:14.736073 25701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:14.740195 25701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:14.740264 25701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:14.740307 25701 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:54:14.740350 25701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:14.745771 25701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:14.746891 25701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:14.747905 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:14.757812 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:14.757859 25707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:14.757825 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:14.758268 25701 server_base.cc:1034] running on GCE node
I20260812 18:54:14.758738 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:14.759622 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:14.760864 25701 hybrid_clock.cc:648] HybridClock initialized: now 1786560854760840 us; error 39 us; skew 500 ppm
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:14.763862 25701 init.cc:377] Logged in from keytab as oryx/127.16.40.194@KRBTEST.COM (short username oryx)
I20260812 18:54:14.764837 25701 webserver.cc:460] Webserver started at http://127.16.40.194:36905/ using document root <none> and password file <none>
I20260812 18:54:14.765399 25701 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:14.765460 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:14.765643 25701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:14.767755 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "53894279daf142d084be533d14e021d2"
server_key_iv: "960ed1aa44454c3aff4342769cb82765"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.768309 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "53894279daf142d084be533d14e021d2"
server_key_iv: "960ed1aa44454c3aff4342769cb82765"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.772475 25701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:54:14.775270 25717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:14.777060 25701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 18:54:14.777212 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
format_stamp: "Formatted at 2026-08-12 18:54:14 on dist-test-slave-rnrw"
server_key: "53894279daf142d084be533d14e021d2"
server_key_iv: "960ed1aa44454c3aff4342769cb82765"
server_key_version: "encryptionkey@0"
I20260812 18:54:14.777351 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:14.793853 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:14.798062 25701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:14.798282 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:14.800154 25701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:14.800226 25701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:14.800280 25701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:14.800309 25701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:54:14 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:14.812145 25701 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:42051
I20260812 18:54:14.812249 25852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:42051 every 8 connection(s)
I20260812 18:54:14.813410 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:14.815739 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.801028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:34755 (local address 127.16.40.254:37101)
0812 18:54:14.801169 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:14.801173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:14.802354 (+ 1181us) server_negotiation.cc:407] Connection header received
0812 18:54:14.803841 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:14.803845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:14.803930 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:14.804059 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:14.805693 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.806250 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.807465 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.807686 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.807803 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.811433 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:14.811440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:14.811446 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:14.811484 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:14.813429 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.814061 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.814065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.814066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.814171 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.814568 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.814572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.814574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.814797 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:14.814912 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:14.815429 (+ 517us) server_negotiation.cc:299] Negotiation successful
0812 18:54:14.815562 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:14.816530 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.801591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.194:34755)
0812 18:54:14.802146 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:14.802214 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:14.803488 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:14.804375 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:14.804382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:14.804845 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:14.805530 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.805552 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.806383 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.806386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.807304 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.807314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.807867 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.807871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.808067 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.808791 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:14.808813 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:14.811230 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:14.813579 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.813585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.813602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.813923 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.814281 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.814284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.814285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.814441 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.814912 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:14.814920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:14.815181 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 18:54:14.815369 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 18:54:14.815715 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260812 18:54:14.817986 25853 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37101
I20260812 18:54:14.818150 25853 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:14.818691 25853 heartbeater.cc:511] Master 127.16.40.254:37101 requested a full tablet report, sending...
I20260812 18:54:14.819927 25445 ts_manager.cc:194] Registered new tserver with Master: 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051)
I20260812 18:54:14.820781 25445 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:34755
I20260812 18:54:14.821228 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 25701
I20260812 18:54:14.821350 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:14.821626 16547 external_mini_cluster.cc:1428] Setting key 79a36853f0db68faae9479173eca0bf8
I20260812 18:54:14.829593 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.820335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:48733 (local address 127.16.40.254:37101)
0812 18:54:14.820576 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:14.820581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:14.820721 (+ 140us) server_negotiation.cc:407] Connection header received
0812 18:54:14.820834 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:14.820836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:14.820878 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:14.820994 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:14.822437 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.823154 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.823985 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.824220 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.824328 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.825772 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:14.825775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:14.825777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:14.825819 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:14.827494 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.828046 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.828049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.828050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.828139 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:14.828583 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:14.828588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:14.828590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:14.828850 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:14.828943 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:14.829328 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 18:54:14.829457 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:54:14.829770 25861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:14.820282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.194:48733)
0812 18:54:14.820610 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:14.820625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:14.820729 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:14.821950 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:14.821953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:14.822168 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:14.822307 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.822314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.823275 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.823279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.823850 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:14.823858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:14.824336 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:14.824339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:14.824466 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:14.825058 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:14.825079 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:14.825617 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:14.827604 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.827610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.827615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.827914 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.828250 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:14.828257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:14.828262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:14.828378 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:14.828963 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:14.828968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:14.829101 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 18:54:14.829243 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 18:54:14.829454 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for oryx/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
I20260812 18:54:14.857474 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:37101
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:14.986181 25864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:14.986475 25864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:14.986523 25864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:14.992069 25864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:14.992139 25864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:14.992182 25864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 18:54:14.992261 25864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:14.998046 25864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.40.254:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:14.999333 25864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:15.000396 25864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:15.010583 25870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:15.010593 25872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:15.010593 25869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:15.010936 25864 server_base.cc:1034] running on GCE node
I20260812 18:54:15.011394 25864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:15.012068 25864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:15.013243 25864 hybrid_clock.cc:648] HybridClock initialized: now 1786560855013186 us; error 75 us; skew 500 ppm
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560855, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:15.016024 25864 init.cc:377] Logged in from keytab as oryx/127.16.40.195@KRBTEST.COM (short username oryx)
I20260812 18:54:15.017040 25864 webserver.cc:460] Webserver started at http://127.16.40.195:39125/ using document root <none> and password file <none>
I20260812 18:54:15.017618 25864 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:15.017678 25864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:15.017851 25864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:15.020016 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "8064dcb5f95f4adbab8aa82e6c463171"
format_stamp: "Formatted at 2026-08-12 18:54:15 on dist-test-slave-rnrw"
server_key: "b03acbf39738efa2731e6687c17a995c"
server_key_iv: "624bced2b41f415c14e0547ae0b0a3cd"
server_key_version: "encryptionkey@0"
I20260812 18:54:15.020586 25864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "8064dcb5f95f4adbab8aa82e6c463171"
format_stamp: "Formatted at 2026-08-12 18:54:15 on dist-test-slave-rnrw"
server_key: "b03acbf39738efa2731e6687c17a995c"
server_key_iv: "624bced2b41f415c14e0547ae0b0a3cd"
server_key_version: "encryptionkey@0"
I20260812 18:54:15.024713 25864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:15.027642 25879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:15.029043 25864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:15.029170 25864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "8064dcb5f95f4adbab8aa82e6c463171"
format_stamp: "Formatted at 2026-08-12 18:54:15 on dist-test-slave-rnrw"
server_key: "b03acbf39738efa2731e6687c17a995c"
server_key_iv: "624bced2b41f415c14e0547ae0b0a3cd"
server_key_version: "encryptionkey@0"
I20260812 18:54:15.029287 25864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:15.053493 25864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:15.057672 25864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:15.057894 25864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:15.059566 25864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:15.059623 25864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:15.059666 25864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:15.059711 25864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560855, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:15.071695 25864 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:38865
I20260812 18:54:15.071981 26014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:38865 every 8 connection(s)
I20260812 18:54:15.072877 25864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:15.075351 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.060355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:42923 (local address 127.16.40.254:37101)
0812 18:54:15.060615 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.060621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.061662 (+ 1041us) server_negotiation.cc:407] Connection header received
0812 18:54:15.062998 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.063004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.063092 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.063203 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:15.064622 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.065202 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.066531 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.066754 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.066866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.070850 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:15.070855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:15.070857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:15.070889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:15.072908 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.073665 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.073672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.073675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.073783 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.074165 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.074168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.074170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.074387 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:15.074505 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.075032 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.075172 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:15.076188 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.060899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.195:42923)
0812 18:54:15.061458 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.061523 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.062726 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.063388 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.063395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.063769 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:15.064411 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.064433 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.065335 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.065339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.066400 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.066410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.066917 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.066921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.067143 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.067895 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:15.067913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:15.070600 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:15.073064 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.073071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.073087 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.073508 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.073896 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.073899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.073901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.074053 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.074505 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:15.074512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:15.074770 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.074962 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.075315 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":120,"threads_started":1}
I20260812 18:54:15.076526 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 25864
I20260812 18:54:15.076699 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:15.077126 16547 external_mini_cluster.cc:1428] Setting key 9a10e1d9bd12c58859344cadeb50b376
I20260812 18:54:15.077867 26015 heartbeater.cc:347] Connected to a master server at 127.16.40.254:37101
I20260812 18:54:15.078076 26015 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:15.078795 26015 heartbeater.cc:511] Master 127.16.40.254:37101 requested a full tablet report, sending...
I20260812 18:54:15.080014 25442 ts_manager.cc:194] Registered new tserver with Master: 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:15.080858 25442 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:42923
I20260812 18:54:15.088898 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.080442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.16.40.195:57409)
0812 18:54:15.080753 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.080773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.080863 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.081102 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.081108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.081362 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:15.081529 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.081537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.082353 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.082357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.083160 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.083171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.083766 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.083769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.083858 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.084405 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:15.084417 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:15.085032 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:15.086963 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.086967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.086969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.087189 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.087717 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.087722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.087723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.087823 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.088284 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:15.088289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:15.088402 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.088504 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.088725 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":141,"threads_started":1}
I20260812 18:54:15.088922 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.080509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:57409 (local address 127.16.40.254:37101)
0812 18:54:15.080713 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.080718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.080841 (+ 123us) server_negotiation.cc:407] Connection header received
0812 18:54:15.080934 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.080936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.080988 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.081076 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:15.081661 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.082222 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.083333 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.083617 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.083720 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.085202 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:15.085206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:15.085209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:15.085238 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:15.086826 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.087311 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.087314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.087315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.087594 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.087964 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.087968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.087970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.088172 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:15.088261 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.088616 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.088737 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 18:54:15.091889 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 18:54:15.120605 25505 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:15.113033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.16.40.254:37101)
0812 18:54:15.113189 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.113193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.113208 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:15.113253 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.113255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.113298 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.113377 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:15.114295 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.114893 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.115869 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.116093 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.116182 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.120302 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55968: BlockingRecv error: recv got EOF from 127.0.0.1:55968 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:15.131485 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.122325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.16.40.254:37101)
0812 18:54:15.122464 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.122469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.122613 (+ 144us) server_negotiation.cc:407] Connection header received
0812 18:54:15.122693 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.122696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.122746 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.122832 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:15.123561 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.124114 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.125063 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.125289 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.125421 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.128125 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:15.128129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:15.128131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:15.128161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:15.129696 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.130164 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.130168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.130176 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.130260 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.130556 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.130560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.130562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.130766 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:15.130908 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.131179 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.131303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:15.135085 25442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:15.138265 25442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:15.153633 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.148096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.16.40.193:45489)
0812 18:54:15.148775 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.148781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.148804 (+ 23us) server_negotiation.cc:407] Connection header received
0812 18:54:15.148881 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.148888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.149097 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.149265 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.149508 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.150259 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.152176 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.152867 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.152997 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.153173 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.153341 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.153462 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":111,"threads_started":1}
I20260812 18:54:15.153662 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.148024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38865 (local address 127.0.0.1:50944)
0812 18:54:15.148428 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.148440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.148513 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.149413 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.149416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.149433 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.149547 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.149553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.151011 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.151018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.152109 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.152121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.153179 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.153183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.153302 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.153325 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.153390 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.153512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":22,"thread_start_us":97,"threads_started":1}
I20260812 18:54:15.153679 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.147567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:45489 (local address 127.0.0.1:37550)
0812 18:54:15.148343 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.148358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.148460 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.149245 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.149247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.149267 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.149366 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.149375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.150385 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.150392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.152064 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.152073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.153004 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.153007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.153123 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.153164 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.153305 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.153504 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":227,"thread_start_us":99,"threads_started":1}
I20260812 18:54:15.154394 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.147286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42051 (local address 127.0.0.1:55748)
0812 18:54:15.147831 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.147885 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.148028 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.148915 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.148922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.148989 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.149151 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.149158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.150787 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.150790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.151984 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.151995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.153179 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.153183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.154039 (+ 856us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.154063 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.154137 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.154224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":752,"thread_start_us":111,"threads_started":1}
I20260812 18:54:15.154457 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.147537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.16.40.194:42051)
0812 18:54:15.148281 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.148294 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.148319 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:54:15.148414 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.148420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.148707 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.149326 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.149479 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.150658 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.152128 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.153029 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.153291 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.153467 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.154198 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.154304 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":139,"threads_started":1}
I20260812 18:54:15.155182 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.148483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.16.40.195:38865)
0812 18:54:15.148932 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.148945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.148967 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:54:15.149049 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.149054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.149255 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.149697 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.149838 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.150896 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.152244 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.153041 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.154699 (+ 1658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.154877 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.154950 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.155059 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":136,"threads_started":1}
I20260812 18:54:15.157405 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 (DEFAULT_TABLE table=test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.157550 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 (DEFAULT_TABLE table=test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.157706 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 (DEFAULT_TABLE table=test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.158502 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cae5c279ff42afa8ec373e4d43c043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.158631 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cae5c279ff42afa8ec373e4d43c043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.158631 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83cae5c279ff42afa8ec373e4d43c043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.165743 26052 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Bootstrap starting.
I20260812 18:54:15.167011 26053 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Bootstrap starting.
I20260812 18:54:15.168102 26052 tablet_bootstrap.cc:654] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.168923 26054 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Bootstrap starting.
I20260812 18:54:15.168960 26052 log.cc:824] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:15.169028 26053 tablet_bootstrap.cc:654] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.169963 26053 log.cc:824] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:15.171190 26054 tablet_bootstrap.cc:654] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.171303 26052 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: No bootstrap required, opened a new log
I20260812 18:54:15.171526 26052 ts_tablet_manager.cc:1397] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:54:15.172009 26054 log.cc:824] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:15.172209 26053 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: No bootstrap required, opened a new log
I20260812 18:54:15.172407 26053 ts_tablet_manager.cc:1397] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:54:15.174117 26054 tablet_bootstrap.cc:492] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: No bootstrap required, opened a new log
I20260812 18:54:15.174281 26054 ts_tablet_manager.cc:1397] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:54:15.175839 26052 raft_consensus.cc:348] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.176151 26052 raft_consensus.cc:374] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.176216 26052 raft_consensus.cc:729] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.176748 26052 consensus_queue.cc:260] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.177270 26053 raft_consensus.cc:348] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.177659 26053 raft_consensus.cc:374] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.177752 26053 raft_consensus.cc:729] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.177810 26054 raft_consensus.cc:348] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.178131 26052 ts_tablet_manager.cc:1428] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:54:15.178130 26054 raft_consensus.cc:374] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.178218 26054 raft_consensus.cc:729] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.178283 26053 consensus_queue.cc:260] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.178598 25853 heartbeater.cc:503] Master 127.16.40.254:37101 was elected leader, sending a full tablet report...
I20260812 18:54:15.178768 26054 consensus_queue.cc:260] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.178987 25691 heartbeater.cc:503] Master 127.16.40.254:37101 was elected leader, sending a full tablet report...
I20260812 18:54:15.179512 26015 heartbeater.cc:503] Master 127.16.40.254:37101 was elected leader, sending a full tablet report...
I20260812 18:54:15.179884 26054 ts_tablet_manager.cc:1428] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:54:15.182119 26053 ts_tablet_manager.cc:1428] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260812 18:54:15.315007 25854 tablet.cc:2369] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:54:15.323705 25692 tablet.cc:2369] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:54:15.324055 26016 tablet.cc:2369] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:54:15.404528 26061 raft_consensus.cc:482] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:15.404821 26061 raft_consensus.cc:504] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.406178 26061 leader_election.cc:288] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051), e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489)
I20260812 18:54:15.410323 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.406434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42051 (local address 127.16.40.195:43727)
0812 18:54:15.406592 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.406611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.406722 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.407084 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.407087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.407111 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.407237 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.407243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.408312 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.408318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.409186 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.409196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.409853 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.409857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.409977 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.409996 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.410086 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.410168 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 18:54:15.410323 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.406670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43727 (local address 127.16.40.194:42051)
0812 18:54:15.406811 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.406818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.406836 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:54:15.406896 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.406899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.406963 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.407057 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.407374 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.408197 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.409321 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.409727 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.409859 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.409919 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.410105 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.410181 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:54:15.410863 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.406625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:45489 (local address 127.16.40.195:36273)
0812 18:54:15.406955 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.406969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.407066 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.407556 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.407559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.407581 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.408081 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.408092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.408940 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.408945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.409714 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.409723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.410425 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.410429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.410539 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.410567 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.410650 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.410723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260812 18:54:15.410903 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.406675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:36273 (local address 127.16.40.193:45489)
0812 18:54:15.406835 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.406840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.407345 (+ 505us) server_negotiation.cc:407] Connection header received
0812 18:54:15.407412 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.407414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.407456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.407516 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.408227 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.408821 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.409849 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.410283 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.410425 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.410477 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.410676 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.410768 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:54:15.411173 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cae5c279ff42afa8ec373e4d43c043" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" is_pre_election: true
I20260812 18:54:15.411523 25800 raft_consensus.cc:2454] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 0.
I20260812 18:54:15.411548 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cae5c279ff42afa8ec373e4d43c043" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3" is_pre_election: true
I20260812 18:54:15.411827 25634 raft_consensus.cc:2454] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 0.
I20260812 18:54:15.412094 25880 leader_election.cc:302] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:15.412398 26061 raft_consensus.cc:2781] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:15.412506 26061 raft_consensus.cc:482] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:15.412590 26061 raft_consensus.cc:3037] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.413676 26061 raft_consensus.cc:504] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.414054 26061 leader_election.cc:288] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051), e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489)
I20260812 18:54:15.414474 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cae5c279ff42afa8ec373e4d43c043" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3"
I20260812 18:54:15.414469 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83cae5c279ff42afa8ec373e4d43c043" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
I20260812 18:54:15.414587 25634 raft_consensus.cc:3037] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.414644 25800 raft_consensus.cc:3037] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.415730 25800 raft_consensus.cc:2454] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 1.
I20260812 18:54:15.415736 25634 raft_consensus.cc:2454] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 1.
I20260812 18:54:15.416075 25880 leader_election.cc:302] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:15.416271 26061 raft_consensus.cc:2781] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:15.416471 26061 raft_consensus.cc:686] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 LEADER]: Becoming Leader. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Running, Role: LEADER
I20260812 18:54:15.416869 26061 consensus_queue.cc:237] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.420359 25445 catalog_manager.cc:5495] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } health_report { overall_health: HEALTHY } } }
I20260812 18:54:15.449913 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.445967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.16.40.195:38865)
0812 18:54:15.446121 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.446127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.446147 (+ 20us) server_negotiation.cc:407] Connection header received
0812 18:54:15.446290 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.446293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.446353 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.446452 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:15.446788 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.447284 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.448273 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.448502 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.448684 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.448965 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:15.449447 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:15.449537 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.449699 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.449760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:54:15.456420 25800 raft_consensus.cc:1261] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Refusing update from remote peer 8064dcb5f95f4adbab8aa82e6c463171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:15.456420 25634 raft_consensus.cc:1261] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Refusing update from remote peer 8064dcb5f95f4adbab8aa82e6c463171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:15.457351 26064 consensus_queue.cc:1035] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:15.457628 26061 consensus_queue.cc:1035] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:15.466223 26068 mvcc.cc:205] Tried to move back new op lower bound from 7317753263941341184 to 7317753263790342144. Current Snapshot: MvccSnapshot[applied={T|T < 7317753263941341184}]
I20260812 18:54:15.466341 26069 mvcc.cc:205] Tried to move back new op lower bound from 7317753263941341184 to 7317753263790342144. Current Snapshot: MvccSnapshot[applied={T|T < 7317753263941341184}]
I20260812 18:54:15.475992 25445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:15.476233 25445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:15.479709 25445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Sending DeleteTablet for 3 replicas of tablet 83cae5c279ff42afa8ec373e4d43c043
I20260812 18:54:15.480801 25939 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:15 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 18:54:15.480801 25776 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:15 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55748
I20260812 18:54:15.480845 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 83cae5c279ff42afa8ec373e4d43c043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:15 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260812 18:54:15.481401 26077 tablet_replica.cc:330] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: stopping tablet replica
I20260812 18:54:15.481443 26079 tablet_replica.cc:330] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: stopping tablet replica
I20260812 18:54:15.481845 26079 raft_consensus.cc:2229] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:15.481904 26077 raft_consensus.cc:2229] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:15.482278 26079 raft_consensus.cc:2258] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:15.482441 26077 raft_consensus.cc:2258] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:15.482406 25445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:15.482963 25445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:15.484292 26079 ts_tablet_manager.cc:1905] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:15.484333 26077 ts_tablet_manager.cc:1905] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:15.484654 26078 tablet_replica.cc:330] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: stopping tablet replica
I20260812 18:54:15.487831 26077 ts_tablet_manager.cc:1918] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:15.487974 26079 ts_tablet_manager.cc:1918] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:15.488114 26079 log.cc:1196] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/83cae5c279ff42afa8ec373e4d43c043
I20260812 18:54:15.488121 26077 log.cc:1196] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/83cae5c279ff42afa8ec373e4d43c043
I20260812 18:54:15.488593 26079 ts_tablet_manager.cc:1939] T 83cae5c279ff42afa8ec373e4d43c043 P e610a259a4d44bf7954e42e14f7510e3: Deleting consensus metadata
I20260812 18:54:15.489120 26077 ts_tablet_manager.cc:1939] T 83cae5c279ff42afa8ec373e4d43c043 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting consensus metadata
I20260812 18:54:15.490513 25426 catalog_manager.cc:4841] TS 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051): tablet 83cae5c279ff42afa8ec373e4d43c043 (table test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]) successfully deleted
I20260812 18:54:15.490805 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 8783001d926c4165a3edf8a31df18599 (DEFAULT_TABLE table=test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.490806 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 8783001d926c4165a3edf8a31df18599 (DEFAULT_TABLE table=test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.491120 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8783001d926c4165a3edf8a31df18599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.491120 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8783001d926c4165a3edf8a31df18599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.493038 25427 catalog_manager.cc:4841] TS e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489): tablet 83cae5c279ff42afa8ec373e4d43c043 (table test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]) successfully deleted
I20260812 18:54:15.493636 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 8783001d926c4165a3edf8a31df18599 (DEFAULT_TABLE table=test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:15.493788 26052 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Bootstrap starting.
I20260812 18:54:15.493927 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8783001d926c4165a3edf8a31df18599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.494673 26052 tablet_bootstrap.cc:654] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.496531 26053 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Bootstrap starting.
I20260812 18:54:15.496654 26054 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Bootstrap starting.
I20260812 18:54:15.497684 26053 tablet_bootstrap.cc:654] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.497738 26054 tablet_bootstrap.cc:654] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.497977 26052 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: No bootstrap required, opened a new log
I20260812 18:54:15.498076 26052 ts_tablet_manager.cc:1397] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 18:54:15.498652 26052 raft_consensus.cc:348] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.498808 26052 raft_consensus.cc:374] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.498859 26052 raft_consensus.cc:729] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.498991 26052 consensus_queue.cc:260] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.499564 26052 ts_tablet_manager.cc:1428] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 18:54:15.499545 26078 raft_consensus.cc:2229] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:15.500097 26078 raft_consensus.cc:2258] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:15.501574 26054 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: No bootstrap required, opened a new log
I20260812 18:54:15.501684 26054 ts_tablet_manager.cc:1397] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260812 18:54:15.502192 26054 raft_consensus.cc:348] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.502322 26054 raft_consensus.cc:374] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.502483 26054 raft_consensus.cc:729] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.502614 26054 consensus_queue.cc:260] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.502962 26054 ts_tablet_manager.cc:1428] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 18:54:15.502303 26078 ts_tablet_manager.cc:1905] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:15.505519 26053 tablet_bootstrap.cc:492] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: No bootstrap required, opened a new log
I20260812 18:54:15.505671 26053 ts_tablet_manager.cc:1397] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260812 18:54:15.506186 26053 raft_consensus.cc:348] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.506311 26053 raft_consensus.cc:374] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.506343 26053 raft_consensus.cc:729] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.506494 26053 consensus_queue.cc:260] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.506661 26078 ts_tablet_manager.cc:1918] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:15.506759 26078 log.cc:1196] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/83cae5c279ff42afa8ec373e4d43c043
I20260812 18:54:15.506799 26053 ts_tablet_manager.cc:1428] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 18:54:15.507164 26078 ts_tablet_manager.cc:1939] T 83cae5c279ff42afa8ec373e4d43c043 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting consensus metadata
I20260812 18:54:15.508486 25426 catalog_manager.cc:4841] TS 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865): tablet 83cae5c279ff42afa8ec373e4d43c043 (table test-table [id=10eec12c4c3643b3999faa3fa3cc2ac5]) successfully deleted
I20260812 18:54:15.610980 26058 raft_consensus.cc:482] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:15.611145 26058 raft_consensus.cc:504] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.612419 26058 leader_election.cc:288] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:15.617473 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.612825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:59673 (local address 127.16.40.195:38865)
0812 18:54:15.612979 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.612984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.613395 (+ 411us) server_negotiation.cc:407] Connection header received
0812 18:54:15.613662 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.613666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.613723 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.613820 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.614097 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.614732 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.616175 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.616839 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.616976 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.617049 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.617260 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.617335 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:15.617502 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.613021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38865 (local address 127.16.40.194:59673)
0812 18:54:15.613286 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.613303 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.613531 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.613840 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.613842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.613859 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.613982 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.613987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.614852 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.614855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.616032 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.616043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.617001 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.617006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.617107 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.617141 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.617238 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.617315 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"spinlock_wait_cycles":14720,"thread_start_us":105,"threads_started":1}
I20260812 18:54:15.617654 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.612848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:50509 (local address 127.16.40.193:45489)
0812 18:54:15.613009 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.613014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.613494 (+ 480us) server_negotiation.cc:407] Connection header received
0812 18:54:15.613970 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.613973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.614040 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.614115 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:15.614385 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.615138 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.616441 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.616924 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.617049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.617106 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.617432 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.617520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:15.618165 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.613116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:45489 (local address 127.16.40.194:50509)
0812 18:54:15.613391 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.613405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.613531 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.614139 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.614141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.614160 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:15.614263 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.614270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.615265 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.615270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.616322 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.616330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.617140 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.617144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.617248 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.617284 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.617925 (+ 641us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.618015 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"spinlock_wait_cycles":25600,"thread_start_us":108,"threads_started":1}
I20260812 18:54:15.618245 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8783001d926c4165a3edf8a31df18599" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171" is_pre_election: true
I20260812 18:54:15.618537 25965 raft_consensus.cc:2454] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 0.
I20260812 18:54:15.618644 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8783001d926c4165a3edf8a31df18599" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3" is_pre_election: true
I20260812 18:54:15.618788 25634 raft_consensus.cc:2454] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 0.
I20260812 18:54:15.618986 25718 leader_election.cc:302] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:15.619288 26058 raft_consensus.cc:2781] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:15.619369 26058 raft_consensus.cc:482] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:15.619422 26058 raft_consensus.cc:3037] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.620961 26058 raft_consensus.cc:504] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.621392 26058 leader_election.cc:288] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 election: Requested vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:15.621824 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8783001d926c4165a3edf8a31df18599" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171"
I20260812 18:54:15.621807 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8783001d926c4165a3edf8a31df18599" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3"
I20260812 18:54:15.621937 25965 raft_consensus.cc:3037] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.621937 25634 raft_consensus.cc:3037] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.623150 25634 raft_consensus.cc:2454] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 1.
I20260812 18:54:15.623153 25965 raft_consensus.cc:2454] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 1.
I20260812 18:54:15.623502 25719 leader_election.cc:302] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, e610a259a4d44bf7954e42e14f7510e3; no voters:
I20260812 18:54:15.623695 26058 raft_consensus.cc:2781] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:15.623921 26058 raft_consensus.cc:686] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 LEADER]: Becoming Leader. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Running, Role: LEADER
I20260812 18:54:15.624185 26058 consensus_queue.cc:237] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.626991 25444 catalog_manager.cc:5495] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194). New cstate: current_term: 1 leader_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } health_report { overall_health: UNKNOWN } } }
Aug 12 18:54:15 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560854, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.254@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:15.687650 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.678871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:37101 (local address 127.0.0.1:55982)
0812 18:54:15.679209 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:15.679227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:15.679349 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:15.679657 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:15.679660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:15.680063 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:15.680166 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.680172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.680960 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.680963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.681556 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:15.681565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.682024 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.682026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:15.682130 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.682724 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:15.682736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:15.684248 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:15.686055 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.686065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.686070 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.686298 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.686569 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:15.686572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:15.686573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:15.686670 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:15.687105 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:15.687111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:15.687225 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 18:54:15.687329 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 18:54:15.687477 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":165,"threads_started":1}
I20260812 18:54:15.687712 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:15.678890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.16.40.254:37101)
0812 18:54:15.679175 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:15.679182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:15.679420 (+ 238us) server_negotiation.cc:407] Connection header received
0812 18:54:15.679473 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:15.679475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:15.679519 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:15.679618 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:15.680293 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.680866 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.681675 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:15.681914 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:15.682007 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:15.684389 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:15.684392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:15.684394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:15.684416 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:15.685945 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.686383 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.686385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.686387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.686466 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:15.686762 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:15.686765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:15.686766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:15.686975 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:15.687062 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:15.687385 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 18:54:15.687550 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":209,"thread_start_us":130,"threads_started":1}
I20260812 18:54:15.692102 25444 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 18:54:15.692988 25444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:15.698215 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 49697cd14b454efa85240413d2637975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fa7683cecb34232b220725712c2d882]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:54:15.698189 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 49697cd14b454efa85240413d2637975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fa7683cecb34232b220725712c2d882]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:54:15.698181 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 49697cd14b454efa85240413d2637975 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6fa7683cecb34232b220725712c2d882]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 18:54:15.698643 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49697cd14b454efa85240413d2637975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.698644 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49697cd14b454efa85240413d2637975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.698652 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49697cd14b454efa85240413d2637975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:15.701596 26054 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: Bootstrap starting.
I20260812 18:54:15.701632 26053 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: Bootstrap starting.
I20260812 18:54:15.701673 26052 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: Bootstrap starting.
I20260812 18:54:15.702589 26052 tablet_bootstrap.cc:654] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.702589 26054 tablet_bootstrap.cc:654] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.702567 26053 tablet_bootstrap.cc:654] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:15.704059 26054 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: No bootstrap required, opened a new log
I20260812 18:54:15.704197 26052 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: No bootstrap required, opened a new log
I20260812 18:54:15.704181 26054 ts_tablet_manager.cc:1397] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 18:54:15.704218 26053 tablet_bootstrap.cc:492] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: No bootstrap required, opened a new log
I20260812 18:54:15.704291 26052 ts_tablet_manager.cc:1397] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:15.704325 26053 ts_tablet_manager.cc:1397] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:15.704753 26054 raft_consensus.cc:348] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.704753 26052 raft_consensus.cc:348] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.704891 26054 raft_consensus.cc:374] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.704891 26052 raft_consensus.cc:374] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.704934 26052 raft_consensus.cc:729] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.704934 26054 raft_consensus.cc:729] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.704897 26053 raft_consensus.cc:348] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.705024 26053 raft_consensus.cc:374] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:15.705062 26053 raft_consensus.cc:729] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Initialized, Role: FOLLOWER
I20260812 18:54:15.705080 26052 consensus_queue.cc:260] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.705102 26054 consensus_queue.cc:260] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.705210 26053 consensus_queue.cc:260] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.705406 26052 ts_tablet_manager.cc:1428] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:15.705353 26072 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:15.705505 26072 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:15.705525 26053 ts_tablet_manager.cc:1428] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 18:54:15.705632 26058 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:15.705747 26058 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:15.705744 26054 ts_tablet_manager.cc:1428] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 18:54:15.705730 26060 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:15.705893 26060 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:15.946259 26072 raft_consensus.cc:482] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:15.946470 26072 raft_consensus.cc:504] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.946964 26072 leader_election.cc:288] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051)
I20260812 18:54:15.947486 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49697cd14b454efa85240413d2637975" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" is_pre_election: true
I20260812 18:54:15.947562 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49697cd14b454efa85240413d2637975" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3" is_pre_election: true
I20260812 18:54:15.947660 25800 raft_consensus.cc:2454] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 0.
I20260812 18:54:15.947775 25634 raft_consensus.cc:2454] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 0.
I20260812 18:54:15.948014 25880 leader_election.cc:302] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:15.948222 26072 raft_consensus.cc:2781] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:15.948287 26072 raft_consensus.cc:482] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:15.948313 26072 raft_consensus.cc:3037] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.949455 26072 raft_consensus.cc:504] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.949851 26072 leader_election.cc:288] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 election: Requested vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051)
I20260812 18:54:15.950248 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49697cd14b454efa85240413d2637975" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
I20260812 18:54:15.950248 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49697cd14b454efa85240413d2637975" candidate_uuid: "8064dcb5f95f4adbab8aa82e6c463171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3"
I20260812 18:54:15.950414 25634 raft_consensus.cc:3037] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.950417 25800 raft_consensus.cc:3037] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:15.951390 25634 raft_consensus.cc:2454] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 1.
I20260812 18:54:15.951398 25800 raft_consensus.cc:2454] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064dcb5f95f4adbab8aa82e6c463171 in term 1.
I20260812 18:54:15.951732 25880 leader_election.cc:302] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:15.951921 26072 raft_consensus.cc:2781] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:15.952028 26072 raft_consensus.cc:686] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 LEADER]: Becoming Leader. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Running, Role: LEADER
I20260812 18:54:15.952176 26072 consensus_queue.cc:237] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:15.952873 26061 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: TxnStatusTablet state changed. Reason: New leader 8064dcb5f95f4adbab8aa82e6c463171. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:15.952991 26061 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:54:15.953212 26099 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 18:54:15.953382 26099 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 18:54:15.954103 25442 catalog_manager.cc:5495] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } health_report { overall_health: HEALTHY } } }
I20260812 18:54:16.004976 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.001157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38865 (local address 127.0.0.1:50956)
0812 18:54:16.001366 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.001384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:16.001497 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:16.001791 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:16.001793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:16.001800 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:54:16.001898 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.001902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.002920 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.002924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.003605 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.003615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.004197 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.004201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.004281 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.004336 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.004691 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.004709 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 18:54:16.004774 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 18:54:16.004852 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 18:54:16.004989 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.001317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.16.40.195:38865)
0812 18:54:16.001463 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.001468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.001485 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:54:16.001616 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.001620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.001675 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.001750 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.001995 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.002789 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.003719 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.004054 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.004162 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.004477 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.004597 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.004677 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.004791 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.004851 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:54:16.007879 25800 raft_consensus.cc:1261] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Refusing update from remote peer 8064dcb5f95f4adbab8aa82e6c463171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.007999 25634 raft_consensus.cc:1261] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Refusing update from remote peer 8064dcb5f95f4adbab8aa82e6c463171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.008375 26061 consensus_queue.cc:1035] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.008706 26072 consensus_queue.cc:1035] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.009886 26058 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: TxnStatusTablet state changed. Reason: New leader 8064dcb5f95f4adbab8aa82e6c463171. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.009999 26058 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:16.010099 26060 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: TxnStatusTablet state changed. Reason: New leader 8064dcb5f95f4adbab8aa82e6c463171. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.010224 26060 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:16.011765 26072 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.011879 26072 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:54:16.011999 26102 mvcc.cc:205] Tried to move back new op lower bound from 7317753266203803648 to 7317753265981480960. Current Snapshot: MvccSnapshot[applied={T|T < 7317753266203803648}]
I20260812 18:54:16.013420 26060 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.013537 26060 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P e610a259a4d44bf7954e42e14f7510e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:16.014850 26061 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.014977 26061 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 8064dcb5f95f4adbab8aa82e6c463171: This TxnStatusTablet replica's current role is: LEADER
I20260812 18:54:16.015971 26058 tablet_replica.cc:439] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8064dcb5f95f4adbab8aa82e6c463171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } } }
I20260812 18:54:16.016098 26058 tablet_replica.cc:442] T 49697cd14b454efa85240413d2637975 P 6c5664c45e0b464e9e66e0b99d9781fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 18:54:16.023133 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.018914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.16.40.194:42051)
0812 18:54:16.019206 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.019210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.019223 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:16.019296 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.019298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.019348 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.019447 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.019708 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.020295 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.021569 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.021841 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.021958 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.022218 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.022695 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.022785 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.022880 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.022934 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":62,"threads_started":1}
I20260812 18:54:16.035296 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.031066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:43161 (local address 127.16.40.195:38865)
0812 18:54:16.031213 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.031218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.031473 (+ 255us) server_negotiation.cc:407] Connection header received
0812 18:54:16.031615 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.031619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.031678 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.031767 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.032013 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.032841 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.033943 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.034308 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.034407 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.034826 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.034906 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.034968 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.035130 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.035188 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:54:16.035315 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.030958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38865 (local address 127.16.40.194:43161)
0812 18:54:16.031392 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.031404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:16.031498 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:16.031786 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:16.031788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:16.031796 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:54:16.031910 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.031915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.032945 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.032948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.033813 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.033825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.034495 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.034499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.034597 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.034664 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.034989 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.034996 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:54:16.035066 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 18:54:16.035153 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":138,"threads_started":1}
I20260812 18:54:16.042804 25965 raft_consensus.cc:1261] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Refusing update from remote peer 6c5664c45e0b464e9e66e0b99d9781fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.042887 26058 consensus_queue.cc:1035] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.043291 26058 consensus_queue.cc:1035] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.067468 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.063589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:55689 (local address 127.16.40.194:42051)
0812 18:54:16.063714 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.063719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.063915 (+ 196us) server_negotiation.cc:407] Connection header received
0812 18:54:16.064065 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.064069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.064123 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.064200 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.064461 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.065075 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.066209 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.066480 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.066575 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.066998 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.067078 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.067154 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.067248 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.067305 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 18:54:16.067469 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.063501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42051 (local address 127.16.40.195:55689)
0812 18:54:16.063836 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.063849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:16.063970 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:16.064212 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:16.064215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:16.064222 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 18:54:16.064354 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.064360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.065183 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.065188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.066083 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.066094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.066667 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.066672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.066763 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.066822 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.067170 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.067177 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:54:16.067237 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 18:54:16.067329 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260812 18:54:16.107121 25444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.107321 25444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.109959 25444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Sending DeleteTablet for 3 replicas of tablet 8783001d926c4165a3edf8a31df18599
I20260812 18:54:16.110638 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 8783001d926c4165a3edf8a31df18599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260812 18:54:16.110996 25776 tablet_service.cc:1514] Processing DeleteTablet for tablet 8783001d926c4165a3edf8a31df18599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55748
I20260812 18:54:16.111509 25939 tablet_service.cc:1514] Processing DeleteTablet for tablet 8783001d926c4165a3edf8a31df18599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 18:54:16.111912 26132 tablet_replica.cc:330] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: stopping tablet replica
I20260812 18:54:16.112114 26132 raft_consensus.cc:2229] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.112332 26132 raft_consensus.cc:2258] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.113276 26132 ts_tablet_manager.cc:1905] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.116881 26132 ts_tablet_manager.cc:1918] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:54:16.117096 26132 log.cc:1196] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/8783001d926c4165a3edf8a31df18599
I20260812 18:54:16.117259 26130 tablet_replica.cc:330] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: stopping tablet replica
I20260812 18:54:16.117439 26130 raft_consensus.cc:2229] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.117575 26130 raft_consensus.cc:2258] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.117615 26132 ts_tablet_manager.cc:1939] T 8783001d926c4165a3edf8a31df18599 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting consensus metadata
I20260812 18:54:16.118053 26131 tablet_replica.cc:330] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: stopping tablet replica
I20260812 18:54:16.118199 26131 raft_consensus.cc:2229] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:16.118557 26131 raft_consensus.cc:2258] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.119129 25426 catalog_manager.cc:4841] TS 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865): tablet 8783001d926c4165a3edf8a31df18599 (table test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]) successfully deleted
I20260812 18:54:16.119691 26131 ts_tablet_manager.cc:1905] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.120916 26130 ts_tablet_manager.cc:1905] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.122714 26131 ts_tablet_manager.cc:1918] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:54:16.122809 26131 log.cc:1196] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/8783001d926c4165a3edf8a31df18599
I20260812 18:54:16.123099 26131 ts_tablet_manager.cc:1939] T 8783001d926c4165a3edf8a31df18599 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting consensus metadata
I20260812 18:54:16.124078 25426 catalog_manager.cc:4841] TS 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051): tablet 8783001d926c4165a3edf8a31df18599 (table test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]) successfully deleted
I20260812 18:54:16.124372 26130 ts_tablet_manager.cc:1918] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:54:16.124475 26130 log.cc:1196] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/8783001d926c4165a3edf8a31df18599
I20260812 18:54:16.124769 26130 ts_tablet_manager.cc:1939] T 8783001d926c4165a3edf8a31df18599 P e610a259a4d44bf7954e42e14f7510e3: Deleting consensus metadata
I20260812 18:54:16.125859 25427 catalog_manager.cc:4841] TS e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489): tablet 8783001d926c4165a3edf8a31df18599 (table test-table [id=d651a347b59f45a9a7dee1f44ab30ef8]) successfully deleted
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[25395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 18:54:16.148651 26092 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:16.140997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.16.40.254:37101)
0812 18:54:16.141175 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.141182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.141369 (+ 187us) server_negotiation.cc:407] Connection header received
0812 18:54:16.141527 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.141530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.141581 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.141659 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:16.142639 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.143172 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.144389 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.144692 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.144780 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.148437 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55988: BlockingRecv error: recv got EOF from 127.0.0.1:55988 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[25395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM
I20260812 18:54:16.160599 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.150423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.16.40.254:37101)
0812 18:54:16.150559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.150563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.150705 (+ 142us) server_negotiation.cc:407] Connection header received
0812 18:54:16.150808 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.150812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.150862 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.150953 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:16.151647 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.152459 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.153499 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.153855 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.153972 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.156876 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:16.156881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:16.156884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:16.156918 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:16.158735 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:16.159216 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:16.159222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:16.159225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:16.159320 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:16.159618 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:16.159621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:16.159624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:16.159862 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:16.159995 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.160259 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.160398 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:54:16.163498 25444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:16.164014 25444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:16.168872 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9572d660664f4198aee77c0f4dd768 (DEFAULT_TABLE table=test-table [id=301b969659e149a783d2fb49f19354ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.169034 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9572d660664f4198aee77c0f4dd768 (DEFAULT_TABLE table=test-table [id=301b969659e149a783d2fb49f19354ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.169034 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 9d9572d660664f4198aee77c0f4dd768 (DEFAULT_TABLE table=test-table [id=301b969659e149a783d2fb49f19354ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.169207 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9572d660664f4198aee77c0f4dd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.169327 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9572d660664f4198aee77c0f4dd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.169333 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d9572d660664f4198aee77c0f4dd768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.171720 26053 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Bootstrap starting.
I20260812 18:54:16.171773 26052 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Bootstrap starting.
I20260812 18:54:16.171773 26054 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Bootstrap starting.
I20260812 18:54:16.172724 26053 tablet_bootstrap.cc:654] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.172725 26052 tablet_bootstrap.cc:654] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.172725 26054 tablet_bootstrap.cc:654] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.174256 26054 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: No bootstrap required, opened a new log
I20260812 18:54:16.174347 26054 ts_tablet_manager.cc:1397] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:16.174492 26052 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: No bootstrap required, opened a new log
I20260812 18:54:16.174563 26052 ts_tablet_manager.cc:1397] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:16.174790 26054 raft_consensus.cc:348] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.174898 26054 raft_consensus.cc:374] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.174934 26054 raft_consensus.cc:729] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.174937 26053 tablet_bootstrap.cc:492] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: No bootstrap required, opened a new log
I20260812 18:54:16.175030 26053 ts_tablet_manager.cc:1397] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:16.175005 26052 raft_consensus.cc:348] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.175061 26054 consensus_queue.cc:260] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.175119 26052 raft_consensus.cc:374] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.175153 26052 raft_consensus.cc:729] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.175278 26052 consensus_queue.cc:260] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.175345 26054 ts_tablet_manager.cc:1428] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:16.175434 26053 raft_consensus.cc:348] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.175526 26052 ts_tablet_manager.cc:1428] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:16.175546 26053 raft_consensus.cc:374] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.175576 26053 raft_consensus.cc:729] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.175684 26053 consensus_queue.cc:260] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.175977 26053 ts_tablet_manager.cc:1428] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:16.208616 26060 raft_consensus.cc:482] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:16.208761 26060 raft_consensus.cc:504] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.209887 26060 leader_election.cc:288] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:16.215099 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.210696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:37343 (local address 127.16.40.195:38865)
0812 18:54:16.210830 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.210835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.210975 (+ 140us) server_negotiation.cc:407] Connection header received
0812 18:54:16.211114 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.211117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.211172 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.211260 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:16.212038 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.212711 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.213785 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.214410 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.214566 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.214630 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.214842 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.214948 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:54:16.215224 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.210349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:38905 (local address 127.16.40.194:42051)
0812 18:54:16.210479 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.210485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.210635 (+ 150us) server_negotiation.cc:407] Connection header received
0812 18:54:16.210903 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.210905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.211000 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.211077 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:16.211584 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.212211 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.213868 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.214459 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.214571 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.214630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.214997 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.215104 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:16.215341 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.210573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38865 (local address 127.16.40.193:37343)
0812 18:54:16.210874 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.210889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:16.211002 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:16.211292 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:16.211294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:16.211339 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:16.211891 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.211902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.212841 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.212844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.213642 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.213653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.214564 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.214568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.214686 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.214717 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 18:54:16.214804 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 18:54:16.215160 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"spinlock_wait_cycles":896,"thread_start_us":72,"threads_started":1}
I20260812 18:54:16.215710 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.210187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:42051 (local address 127.16.40.193:38905)
0812 18:54:16.210478 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.210495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:16.210734 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:16.211249 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:16.211268 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:16.211288 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:16.211446 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.211452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.212336 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.212340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.213692 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:16.213706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.214727 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.214730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:16.214842 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.214868 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:54:16.215412 (+ 544us) client_negotiation.cc:240] Negotiation successful
0812 18:54:16.215505 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":16,"spinlock_wait_cycles":13440,"thread_start_us":97,"threads_started":1}
I20260812 18:54:16.216018 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9572d660664f4198aee77c0f4dd768" candidate_uuid: "e610a259a4d44bf7954e42e14f7510e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171" is_pre_election: true
I20260812 18:54:16.216189 25965 raft_consensus.cc:2454] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e610a259a4d44bf7954e42e14f7510e3 in term 0.
I20260812 18:54:16.216152 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9572d660664f4198aee77c0f4dd768" candidate_uuid: "e610a259a4d44bf7954e42e14f7510e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" is_pre_election: true
I20260812 18:54:16.216295 25800 raft_consensus.cc:2454] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e610a259a4d44bf7954e42e14f7510e3 in term 0.
I20260812 18:54:16.216657 25550 leader_election.cc:302] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8064dcb5f95f4adbab8aa82e6c463171, e610a259a4d44bf7954e42e14f7510e3; no voters:
I20260812 18:54:16.217006 26060 raft_consensus.cc:2781] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:16.217082 26060 raft_consensus.cc:482] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:16.217125 26060 raft_consensus.cc:3037] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.218546 26060 raft_consensus.cc:504] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.218973 26060 leader_election.cc:288] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [CANDIDATE]: Term 1 election: Requested vote from peers 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:16.219410 25800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9572d660664f4198aee77c0f4dd768" candidate_uuid: "e610a259a4d44bf7954e42e14f7510e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5664c45e0b464e9e66e0b99d9781fa"
I20260812 18:54:16.219560 25800 raft_consensus.cc:3037] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.219630 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d9572d660664f4198aee77c0f4dd768" candidate_uuid: "e610a259a4d44bf7954e42e14f7510e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171"
I20260812 18:54:16.219796 25965 raft_consensus.cc:3037] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.220891 25965 raft_consensus.cc:2454] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e610a259a4d44bf7954e42e14f7510e3 in term 1.
I20260812 18:54:16.220921 25800 raft_consensus.cc:2454] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e610a259a4d44bf7954e42e14f7510e3 in term 1.
I20260812 18:54:16.221251 25550 leader_election.cc:302] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, e610a259a4d44bf7954e42e14f7510e3; no voters:
I20260812 18:54:16.221565 26060 raft_consensus.cc:2781] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:16.221809 26060 raft_consensus.cc:686] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 1 LEADER]: Becoming Leader. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Running, Role: LEADER
I20260812 18:54:16.222090 26060 consensus_queue.cc:237] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.224953 25442 catalog_manager.cc:5495] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193). New cstate: current_term: 1 leader_uuid: "e610a259a4d44bf7954e42e14f7510e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260812 18:54:16.236235 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.231857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.16.40.193:45489)
0812 18:54:16.232198 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.232208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.232224 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:16.232309 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.232312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.232384 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.232531 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.232968 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.233735 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.234651 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.234905 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.235019 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.235310 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.235790 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.235922 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.235987 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.236058 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
I20260812 18:54:16.239199 25800 raft_consensus.cc:1261] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Refusing update from remote peer e610a259a4d44bf7954e42e14f7510e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.239264 25965 raft_consensus.cc:1261] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Refusing update from remote peer e610a259a4d44bf7954e42e14f7510e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.239758 26060 consensus_queue.cc:1035] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.240209 26060 consensus_queue.cc:1035] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.254789 25442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.254995 25442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.257750 25442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Sending DeleteTablet for 3 replicas of tablet 9d9572d660664f4198aee77c0f4dd768
I20260812 18:54:16.258496 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9572d660664f4198aee77c0f4dd768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260812 18:54:16.258728 26130 tablet_replica.cc:330] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: stopping tablet replica
I20260812 18:54:16.258857 26130 raft_consensus.cc:2229] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:16.258983 25776 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9572d660664f4198aee77c0f4dd768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55748
I20260812 18:54:16.259227 26130 raft_consensus.cc:2258] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.259434 25939 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d9572d660664f4198aee77c0f4dd768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 18:54:16.259745 26131 tablet_replica.cc:330] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: stopping tablet replica
I20260812 18:54:16.259886 26131 raft_consensus.cc:2229] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.260051 26131 raft_consensus.cc:2258] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.260070 26130 ts_tablet_manager.cc:1905] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.260255 26132 tablet_replica.cc:330] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: stopping tablet replica
I20260812 18:54:16.260396 26132 raft_consensus.cc:2229] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.260617 26132 raft_consensus.cc:2258] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.262056 25442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:16.262568 25442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:16.263259 26130 ts_tablet_manager.cc:1918] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:16.263355 26130 log.cc:1196] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/9d9572d660664f4198aee77c0f4dd768
I20260812 18:54:16.263712 26130 ts_tablet_manager.cc:1939] T 9d9572d660664f4198aee77c0f4dd768 P e610a259a4d44bf7954e42e14f7510e3: Deleting consensus metadata
I20260812 18:54:16.264532 25427 catalog_manager.cc:4841] TS e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489): tablet 9d9572d660664f4198aee77c0f4dd768 (table test-table [id=301b969659e149a783d2fb49f19354ed]) successfully deleted
I20260812 18:54:16.265575 26132 ts_tablet_manager.cc:1905] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.268286 26132 ts_tablet_manager.cc:1918] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:16.268368 26132 log.cc:1196] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/9d9572d660664f4198aee77c0f4dd768
I20260812 18:54:16.268666 26132 ts_tablet_manager.cc:1939] T 9d9572d660664f4198aee77c0f4dd768 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting consensus metadata
I20260812 18:54:16.269575 26131 ts_tablet_manager.cc:1905] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.271973 26131 ts_tablet_manager.cc:1918] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:16.272063 26131 log.cc:1196] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/9d9572d660664f4198aee77c0f4dd768
I20260812 18:54:16.272326 26131 ts_tablet_manager.cc:1939] T 9d9572d660664f4198aee77c0f4dd768 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting consensus metadata
I20260812 18:54:16.273068 25426 catalog_manager.cc:4841] TS 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051): tablet 9d9572d660664f4198aee77c0f4dd768 (table test-table [id=301b969659e149a783d2fb49f19354ed]) successfully deleted
I20260812 18:54:16.273241 25426 catalog_manager.cc:4841] TS 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865): tablet 9d9572d660664f4198aee77c0f4dd768 (table test-table [id=301b969659e149a783d2fb49f19354ed]) successfully deleted
I20260812 18:54:16.277454 25776 tablet_service.cc:1467] Processing CreateTablet for tablet 354ac0468ed942d29040225bd0c332c1 (DEFAULT_TABLE table=test-table [id=62cafcad2b994d4e900340c45145a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.277784 25776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354ac0468ed942d29040225bd0c332c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.280352 26052 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Bootstrap starting.
I20260812 18:54:16.281416 26052 tablet_bootstrap.cc:654] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.275700 25939 tablet_service.cc:1467] Processing CreateTablet for tablet 354ac0468ed942d29040225bd0c332c1 (DEFAULT_TABLE table=test-table [id=62cafcad2b994d4e900340c45145a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.282366 25939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354ac0468ed942d29040225bd0c332c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.283455 26052 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: No bootstrap required, opened a new log
I20260812 18:54:16.283550 26052 ts_tablet_manager.cc:1397] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:16.284041 26052 raft_consensus.cc:348] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.284161 26052 raft_consensus.cc:374] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.284194 26052 raft_consensus.cc:729] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.284319 26052 consensus_queue.cc:260] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.284600 26052 ts_tablet_manager.cc:1428] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:16.285794 26054 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Bootstrap starting.
I20260812 18:54:16.286262 25611 tablet_service.cc:1467] Processing CreateTablet for tablet 354ac0468ed942d29040225bd0c332c1 (DEFAULT_TABLE table=test-table [id=62cafcad2b994d4e900340c45145a230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:16.286630 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354ac0468ed942d29040225bd0c332c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:16.286717 26054 tablet_bootstrap.cc:654] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.288015 26054 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: No bootstrap required, opened a new log
I20260812 18:54:16.288115 26054 ts_tablet_manager.cc:1397] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 18:54:16.288632 26054 raft_consensus.cc:348] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.288754 26054 raft_consensus.cc:374] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.288793 26054 raft_consensus.cc:729] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064dcb5f95f4adbab8aa82e6c463171, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.288924 26054 consensus_queue.cc:260] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.289640 26053 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: Bootstrap starting.
I20260812 18:54:16.290614 26053 tablet_bootstrap.cc:654] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:16.290827 26054 ts_tablet_manager.cc:1428] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:16.292369 26053 tablet_bootstrap.cc:492] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: No bootstrap required, opened a new log
I20260812 18:54:16.292476 26053 ts_tablet_manager.cc:1397] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:16.292982 26053 raft_consensus.cc:348] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.293105 26053 raft_consensus.cc:374] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:16.293135 26053 raft_consensus.cc:729] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e610a259a4d44bf7954e42e14f7510e3, State: Initialized, Role: FOLLOWER
I20260812 18:54:16.293254 26053 consensus_queue.cc:260] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.293517 26053 ts_tablet_manager.cc:1428] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:54:16.316882 26085 raft_consensus.cc:482] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:16.317071 26085 raft_consensus.cc:504] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.317677 26085 leader_election.cc:288] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:16.318341 25633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354ac0468ed942d29040225bd0c332c1" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3" is_pre_election: true
I20260812 18:54:16.318365 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354ac0468ed942d29040225bd0c332c1" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171" is_pre_election: true
I20260812 18:54:16.318547 25965 raft_consensus.cc:2454] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 0.
I20260812 18:54:16.318552 25633 raft_consensus.cc:2454] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 0.
I20260812 18:54:16.318905 25718 leader_election.cc:302] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:16.319166 26085 raft_consensus.cc:2781] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:16.319238 26085 raft_consensus.cc:482] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:16.319293 26085 raft_consensus.cc:3037] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.320401 26085 raft_consensus.cc:504] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.320866 26085 leader_election.cc:288] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 election: Requested vote from peers e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489), 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865)
I20260812 18:54:16.321202 25633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354ac0468ed942d29040225bd0c332c1" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e610a259a4d44bf7954e42e14f7510e3"
I20260812 18:54:16.321349 25633 raft_consensus.cc:3037] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.321354 25965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354ac0468ed942d29040225bd0c332c1" candidate_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064dcb5f95f4adbab8aa82e6c463171"
I20260812 18:54:16.321493 25965 raft_consensus.cc:3037] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:16.322486 25965 raft_consensus.cc:2454] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 1.
I20260812 18:54:16.322635 25633 raft_consensus.cc:2454] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c5664c45e0b464e9e66e0b99d9781fa in term 1.
I20260812 18:54:16.322845 25718 leader_election.cc:302] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5664c45e0b464e9e66e0b99d9781fa, 8064dcb5f95f4adbab8aa82e6c463171; no voters:
I20260812 18:54:16.323100 26085 raft_consensus.cc:2781] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:16.323194 26085 raft_consensus.cc:686] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 LEADER]: Becoming Leader. State: Replica: 6c5664c45e0b464e9e66e0b99d9781fa, State: Running, Role: LEADER
I20260812 18:54:16.323380 26085 consensus_queue.cc:237] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } }
I20260812 18:54:16.325340 25442 catalog_manager.cc:5495] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194). New cstate: current_term: 1 leader_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c5664c45e0b464e9e66e0b99d9781fa" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20260812 18:54:16.346585 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.342624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.16.40.194:42051)
0812 18:54:16.342950 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:16.342955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:16.342967 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:54:16.343015 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:16.343017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:16.343082 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:16.343156 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:16.343445 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.344255 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.345313 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:16.345701 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:16.345811 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:16.346097 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:16.346193 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:16.346268 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:16.346360 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 18:54:16.346430 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260812 18:54:16.354982 25633 raft_consensus.cc:1261] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Refusing update from remote peer 6c5664c45e0b464e9e66e0b99d9781fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.355046 25965 raft_consensus.cc:1261] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Refusing update from remote peer 6c5664c45e0b464e9e66e0b99d9781fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:16.355629 26085 consensus_queue.cc:1035] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e610a259a4d44bf7954e42e14f7510e3" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.355829 26058 consensus_queue.cc:1035] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064dcb5f95f4adbab8aa82e6c463171" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:16.394038 25444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.394213 25444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:16.396656 25444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P feb4ce57b74d42519ec42ddd01e86ab2: Sending DeleteTablet for 3 replicas of tablet 354ac0468ed942d29040225bd0c332c1
I20260812 18:54:16.397341 25611 tablet_service.cc:1514] Processing DeleteTablet for tablet 354ac0468ed942d29040225bd0c332c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260812 18:54:16.397399 25776 tablet_service.cc:1514] Processing DeleteTablet for tablet 354ac0468ed942d29040225bd0c332c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55748
I20260812 18:54:16.397590 26130 tablet_replica.cc:330] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3: stopping tablet replica
I20260812 18:54:16.397688 26131 tablet_replica.cc:330] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: stopping tablet replica
I20260812 18:54:16.397754 26130 raft_consensus.cc:2229] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.397826 26131 raft_consensus.cc:2229] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:16.397903 26130 raft_consensus.cc:2258] T 354ac0468ed942d29040225bd0c332c1 P e610a259a4d44bf7954e42e14f7510e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.398051 26131 raft_consensus.cc:2258] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.398423 25939 tablet_service.cc:1514] Processing DeleteTablet for tablet 354ac0468ed942d29040225bd0c332c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:16 UTC) from {username='oryx', principal='oryx/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:50944
I20260812 18:54:16.398633 26132 tablet_replica.cc:330] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: stopping tablet replica
I20260812 18:54:16.398763 26132 raft_consensus.cc:2229] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:16.398821 26131 ts_tablet_manager.cc:1905] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.398878 26132 raft_consensus.cc:2258] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:16.399345 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 25532
I20260812 18:54:16.402606 26131 ts_tablet_manager.cc:1918] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:54:16.402681 26131 log.cc:1196] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/354ac0468ed942d29040225bd0c332c1
I20260812 18:54:16.403028 26131 ts_tablet_manager.cc:1939] T 354ac0468ed942d29040225bd0c332c1 P 6c5664c45e0b464e9e66e0b99d9781fa: Deleting consensus metadata
I20260812 18:54:16.403923 26132 ts_tablet_manager.cc:1905] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:16.403900 25426 catalog_manager.cc:4841] TS 6c5664c45e0b464e9e66e0b99d9781fa (127.16.40.194:42051): tablet 354ac0468ed942d29040225bd0c332c1 (table test-table [id=62cafcad2b994d4e900340c45145a230]) successfully deleted
I20260812 18:54:16.407027 26132 ts_tablet_manager.cc:1918] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 18:54:16.407132 26132 log.cc:1196] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/354ac0468ed942d29040225bd0c332c1
I20260812 18:54:16.407440 26132 ts_tablet_manager.cc:1939] T 354ac0468ed942d29040225bd0c332c1 P 8064dcb5f95f4adbab8aa82e6c463171: Deleting consensus metadata
I20260812 18:54:16.408396 25426 catalog_manager.cc:4841] TS 8064dcb5f95f4adbab8aa82e6c463171 (127.16.40.195:38865): tablet 354ac0468ed942d29040225bd0c332c1 (table test-table [id=62cafcad2b994d4e900340c45145a230]) successfully deleted
W20260812 18:54:16.414040 25427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:45489 (error 108)
I20260812 18:54:16.414245 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 25701
I20260812 18:54:16.418810 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:16.417809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:45489 (local address 127.0.0.1:37574)
0812 18:54:16.418624 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:16.418698 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":89,"threads_started":1}
W20260812 18:54:16.418993 25427 catalog_manager.cc:4584] TS e610a259a4d44bf7954e42e14f7510e3 (127.16.40.193:45489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 354ac0468ed942d29040225bd0c332c1: Network error: Client connection negotiation failed: client connection to 127.16.40.193:45489: connect: Connection refused (error 111)
I20260812 18:54:16.427189 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 25864
I20260812 18:54:16.437662 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 25409
2026-08-12T18:54:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3681 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[26171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[26171](info): set up 2 sockets
Aug 12 18:54:16 dist-test-slave-rnrw krb5kdc[26171](info): commencing operation
krb5kdc: starting...
W20260812 18:54:17.581691 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:17 dist-test-slave-rnrw krb5kdc[26171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:54:17.724069 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45223
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:17.844363 26184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:17.844799 26184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:17.844877 26184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:17.849221 26184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:17.849300 26184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:17.849320 26184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:17.849337 26184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:17.849354 26184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:17.855469 26184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:45223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:17.856829 26184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:17.857959 26184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:17.867362 26191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:17.867368 26193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:17.867368 26190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:17.867777 26184 server_base.cc:1034] running on GCE node
I20260812 18:54:17.868292 26184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:17.869568 26184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:17.870754 26184 hybrid_clock.cc:648] HybridClock initialized: now 1786560857870731 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T18:54:17Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1440 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:17 dist-test-slave-rnrw krb5kdc[26198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:17 dist-test-slave-rnrw krb5kdc[26198](info): set up 2 sockets
Aug 12 18:54:17 dist-test-slave-rnrw krb5kdc[26198](info): commencing operation
krb5kdc: starting...
W20260812 18:54:19.019716 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:54:19.153962 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36241
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:19.272008 26214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:19.272346 26214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:19.272419 26214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:19.276592 26214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:19.276701 26214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:19.276738 26214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:19.276779 26214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:19.276816 26214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:19.282446 26214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:19.283653 26214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:19.284744 26214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:19.293648 26223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.293654 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.293638 26225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:19.293875 26214 server_base.cc:1034] running on GCE node
I20260812 18:54:19.294483 26214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:19.295742 26214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:19.296949 26214 hybrid_clock.cc:648] HybridClock initialized: now 1786560859296936 us; error 29 us; skew 500 ppm
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:19.305281 26214 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:54:19.306341 26214 webserver.cc:460] Webserver started at http://127.16.40.254:42663/ using document root <none> and password file <none>
I20260812 18:54:19.306943 26214 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:19.307009 26214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:19.307277 26214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:19.309626 26214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "74d5019ad7f74dc189ad97c844192606"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "57e945d06da8a96eddc61e0126306bba"
server_key_iv: "c2e524daf0eae93b5546228c01d20bf0"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.310388 26214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "74d5019ad7f74dc189ad97c844192606"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "57e945d06da8a96eddc61e0126306bba"
server_key_iv: "c2e524daf0eae93b5546228c01d20bf0"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.315061 26214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:54:19.317654 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.318997 26214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 18:54:19.319188 26214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "74d5019ad7f74dc189ad97c844192606"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "57e945d06da8a96eddc61e0126306bba"
server_key_iv: "c2e524daf0eae93b5546228c01d20bf0"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.319324 26214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:19.335294 26214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:19.338716 26214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:19.338928 26214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:19.346719 26214 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:36241
I20260812 18:54:19.346740 26293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:36241 every 8 connection(s)
I20260812 18:54:19.347832 26214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:19.350718 26294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:19.351857 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26214
I20260812 18:54:19.352003 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:19.352349 16547 external_mini_cluster.cc:1428] Setting key 7dc36ffa47828344f7ec342b0c1a4190
I20260812 18:54:19.357825 26294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Bootstrap starting.
I20260812 18:54:19.360778 26294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:19.361682 26294 log.cc:824] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:19.364486 26294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: No bootstrap required, opened a new log
I20260812 18:54:19.368281 26294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:19.368533 26294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:19.368695 26294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d5019ad7f74dc189ad97c844192606, State: Initialized, Role: FOLLOWER
I20260812 18:54:19.368899 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:19.353925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.16.40.254:36241)
0812 18:54:19.354399 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:19.354418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:19.354463 (+ 45us) server_negotiation.cc:407] Connection header received
0812 18:54:19.355302 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:19.355325 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:19.355764 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:19.356149 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:19.357047 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.358505 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.359463 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.360172 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.360350 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:19.363469 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:19.363477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:19.363496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:19.363533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:19.365611 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.366124 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.366130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.366135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.366256 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.366572 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.366576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.366578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.367025 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:19.367262 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:19.367625 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 18:54:19.367938 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":129,"threads_started":1}
I20260812 18:54:19.369271 26294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:19.369449 26294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:19.369529 26294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:19.369630 26294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:19.370785 26294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:19.371244 26294 leader_election.cc:302] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74d5019ad7f74dc189ad97c844192606; no voters:
I20260812 18:54:19.371655 26294 leader_election.cc:288] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:19.371743 26301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:19.371934 26301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 LEADER]: Becoming Leader. State: Replica: 74d5019ad7f74dc189ad97c844192606, State: Running, Role: LEADER
I20260812 18:54:19.372267 26301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:19.373114 26294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:19.374219 26304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74d5019ad7f74dc189ad97c844192606. Latest consensus state: current_term: 1 leader_uuid: "74d5019ad7f74dc189ad97c844192606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } } }
I20260812 18:54:19.374264 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74d5019ad7f74dc189ad97c844192606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } } }
I20260812 18:54:19.374486 26304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:19.374521 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:19.375136 26312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:19.378980 26312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:19.385289 26312 catalog_manager.cc:1347] Generated new cluster ID: 91152d354f7c4ecf8d3b0b6c6bd7b770
I20260812 18:54:19.385389 26312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:19.391346 26312 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:19.392350 26312 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:19.398626 26312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Generated new TSK 0
I20260812 18:54:19.399560 26312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:54:19.439868 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:19.566843 26325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:19.567183 26325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:19.567268 26325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:19.571656 26325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:19.571767 26325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:19.571869 26325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:19.577828 26325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:19.579035 26325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:19.580127 26325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:19.590392 26331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.590377 26330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.590377 26333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:19.590893 26325 server_base.cc:1034] running on GCE node
I20260812 18:54:19.591321 26325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:19.591991 26325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:19.593228 26325 hybrid_clock.cc:648] HybridClock initialized: now 1786560859593197 us; error 53 us; skew 500 ppm
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:19.596208 26325 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:54:19.597251 26325 webserver.cc:460] Webserver started at http://127.16.40.193:33431/ using document root <none> and password file <none>
I20260812 18:54:19.597926 26325 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:19.598035 26325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:19.598279 26325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:19.600427 26325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "9b9bb632b0e44332aba1841cd983efdb"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "6d19989bde3371d4d979b106dbbbe151"
server_key_iv: "ce2c2710399647872a169926129e6c18"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.601063 26325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "9b9bb632b0e44332aba1841cd983efdb"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "6d19989bde3371d4d979b106dbbbe151"
server_key_iv: "ce2c2710399647872a169926129e6c18"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.605093 26325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:19.607757 26343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.609207 26325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:19.609359 26325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "9b9bb632b0e44332aba1841cd983efdb"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "6d19989bde3371d4d979b106dbbbe151"
server_key_iv: "ce2c2710399647872a169926129e6c18"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.609486 26325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:19.624940 26325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:19.629192 26325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:19.629364 26325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:19.629914 26325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:19.630752 26325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:19.630800 26325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.630842 26325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:19.630858 26325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.640625 26325 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:45731
I20260812 18:54:19.640751 26474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:45731 every 8 connection(s)
I20260812 18:54:19.641659 26325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:19.648713 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26325
I20260812 18:54:19.648890 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:19.649220 16547 external_mini_cluster.cc:1428] Setting key 4733b2b1f4195bfef3539b2cf191cb7b
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:19.657207 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:19.643774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:37209 (local address 127.16.40.254:36241)
0812 18:54:19.643929 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:19.643933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:19.644981 (+ 1048us) server_negotiation.cc:407] Connection header received
0812 18:54:19.646274 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:19.646278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:19.646331 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:19.646417 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:19.647706 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.648388 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.649874 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.650191 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.650287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:19.653361 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:19.653368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:19.653375 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:19.653401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:19.654888 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.655460 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.655464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.655465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.655588 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.655957 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.655961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.655963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.656165 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:19.656277 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:19.656885 (+ 608us) server_negotiation.cc:299] Negotiation successful
0812 18:54:19.657030 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:54:19.658100 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:19.644197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:37209)
0812 18:54:19.644809 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:19.644857 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:19.646050 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:19.646635 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:19.646642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:19.647029 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:19.647546 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:19.647560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.648610 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.648617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:19.649716 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:19.649729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.650359 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.650363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:19.650615 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:19.651301 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:19.651319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:19.653176 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:19.655029 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:19.655035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:19.655054 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:19.655335 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:19.655690 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:19.655694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:19.655696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:19.655852 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:19.656278 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:19.656287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:19.656615 (+ 328us) client_negotiation.cc:769] Sending connection context
0812 18:54:19.656823 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 18:54:19.657209 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":134,"threads_started":1}
I20260812 18:54:19.659689 26475 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36241
I20260812 18:54:19.659952 26475 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:19.660494 26475 heartbeater.cc:511] Master 127.16.40.254:36241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:54:19.662254 26252 ts_manager.cc:194] Registered new tserver with Master: 9b9bb632b0e44332aba1841cd983efdb (127.16.40.193:45731)
I20260812 18:54:19.663559 26252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:37209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:54:19.681759 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:19.800407 26481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:19.800752 26481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:19.800835 26481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:19.805097 26481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:19.805181 26481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:19.805243 26481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:19.811058 26481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:19.812291 26481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:19.813558 26481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:19.824530 26488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.824617 26490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:19.824518 26486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:19.824957 26481 server_base.cc:1034] running on GCE node
I20260812 18:54:19.825425 26481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:19.826213 26481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:19.827425 26481 hybrid_clock.cc:648] HybridClock initialized: now 1786560859827399 us; error 44 us; skew 500 ppm
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:19.830602 26481 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:54:19.831686 26481 webserver.cc:460] Webserver started at http://127.16.40.194:43805/ using document root <none> and password file <none>
I20260812 18:54:19.832340 26481 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:19.832440 26481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:19.832746 26481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:19.834978 26481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "c8dcec90a1bf4d53a51b8120f34a8ffd"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "9c2141be2170d9b85d6c6ccfa52b961f"
server_key_iv: "bea4c55a09b4f99fe1b658712d45646f"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.835585 26481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "c8dcec90a1bf4d53a51b8120f34a8ffd"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "9c2141be2170d9b85d6c6ccfa52b961f"
server_key_iv: "bea4c55a09b4f99fe1b658712d45646f"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.839743 26481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:54:19.842597 26497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.844022 26481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:19.844206 26481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "c8dcec90a1bf4d53a51b8120f34a8ffd"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "9c2141be2170d9b85d6c6ccfa52b961f"
server_key_iv: "bea4c55a09b4f99fe1b658712d45646f"
server_key_version: "encryptionkey@0"
I20260812 18:54:19.844352 26481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:19.872464 26481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:19.877398 26481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:19.877730 26481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:19.878669 26481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:19.880070 26481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:19.880148 26481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.880218 26481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:19.880259 26481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:19.892988 26481 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:40651
I20260812 18:54:19.893013 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:40651 every 8 connection(s)
I20260812 18:54:19.894079 26481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:19.895150 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26481
I20260812 18:54:19.895303 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:19.895617 16547 external_mini_cluster.cc:1428] Setting key b60b6b940b5af392774646e58f01bc35
Aug 12 18:54:19 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560859, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:54:19.912361 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:19.896322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53709 (local address 127.16.40.254:36241)
0812 18:54:19.896487 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:19.896490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:19.898051 (+ 1561us) server_negotiation.cc:407] Connection header received
0812 18:54:19.899704 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:19.899710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:19.899786 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:19.899903 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:19.901461 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.902317 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.904268 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.904753 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.904909 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:19.908237 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:19.908240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:19.908243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:19.908273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:19.909904 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.910602 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.910607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.910622 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.910733 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:19.911128 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:19.911134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:19.911138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:19.911356 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:19.911476 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:19.912047 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 18:54:19.912189 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:19.913695 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:19.896857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:53709)
0812 18:54:19.897857 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:19.897933 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:19.899323 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:19.900128 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:19.900141 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:19.900696 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:19.901286 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:19.901302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.902500 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.902503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:19.904102 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:19.904116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:19.904970 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:19.904973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:19.905150 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:19.905990 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:19.906010 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:19.908016 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:19.910090 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:19.910097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:19.910113 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:19.910462 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:19.910840 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:19.910844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:19.910846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:19.911021 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:19.911476 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:19.911486 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:19.911787 (+ 301us) client_negotiation.cc:769] Sending connection context
0812 18:54:19.912000 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 18:54:19.912391 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":187,"threads_started":1}
I20260812 18:54:19.915198 26635 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36241
I20260812 18:54:19.915486 26635 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:19.916064 26635 heartbeater.cc:511] Master 127.16.40.254:36241 requested a full tablet report, sending...
I20260812 18:54:19.917348 26252 ts_manager.cc:194] Registered new tserver with Master: c8dcec90a1bf4d53a51b8120f34a8ffd (127.16.40.194:40651)
I20260812 18:54:19.917972 26252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:53709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:54:19.932193 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:20.055022 26641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:20.055382 26641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:20.055469 26641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:20.059746 26641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:20.059875 26641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:20.059975 26641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:20.065706 26641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:20.066941 26641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:20.068099 26641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:20.078506 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.078617 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.078693 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.078949 26641 server_base.cc:1034] running on GCE node
I20260812 18:54:20.079468 26641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:20.080255 26641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:20.081444 26641 hybrid_clock.cc:648] HybridClock initialized: now 1786560860081439 us; error 93 us; skew 500 ppm
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:20.084789 26641 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:54:20.085949 26641 webserver.cc:460] Webserver started at http://127.16.40.195:40861/ using document root <none> and password file <none>
I20260812 18:54:20.086601 26641 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:20.086670 26641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:20.086865 26641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:20.089387 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "2021b23b04094d3c8f654c8c4482e466"
format_stamp: "Formatted at 2026-08-12 18:54:20 on dist-test-slave-rnrw"
server_key: "29fc8399890440c0f1c78e61bfaa40b2"
server_key_iv: "9bcd5e06ee1d6115ce810f68aa7ff841"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.090070 26641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "2021b23b04094d3c8f654c8c4482e466"
format_stamp: "Formatted at 2026-08-12 18:54:20 on dist-test-slave-rnrw"
server_key: "29fc8399890440c0f1c78e61bfaa40b2"
server_key_iv: "9bcd5e06ee1d6115ce810f68aa7ff841"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.095183 26641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 18:54:20.098461 26659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.100054 26641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:20.100231 26641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "2021b23b04094d3c8f654c8c4482e466"
format_stamp: "Formatted at 2026-08-12 18:54:20 on dist-test-slave-rnrw"
server_key: "29fc8399890440c0f1c78e61bfaa40b2"
server_key_iv: "9bcd5e06ee1d6115ce810f68aa7ff841"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.100373 26641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:20.118404 26641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:20.121984 26641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:20.122164 26641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:20.122849 26641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:20.123842 26641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:20.123895 26641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.123960 26641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:20.124004 26641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.134557 26641 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:33285
I20260812 18:54:20.134577 26787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:33285 every 8 connection(s)
I20260812 18:54:20.135567 26641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:20.141045 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26641
I20260812 18:54:20.141215 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:20.141539 16547 external_mini_cluster.cc:1428] Setting key 03d6a9b3a32e6aeadbeda44b95806a98
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:20.152325 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.137699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:60465 (local address 127.16.40.254:36241)
0812 18:54:20.137880 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.137885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.138810 (+ 925us) server_negotiation.cc:407] Connection header received
0812 18:54:20.140317 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.140320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.140375 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.140458 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.142341 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.143034 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.144479 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.144817 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.144939 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.147828 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:20.147832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:20.147834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:20.147864 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:20.149780 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:20.150446 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:20.150453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:20.150460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:20.150571 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:20.150987 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:20.150992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:20.150994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:20.151224 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:20.151356 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:20.151977 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 18:54:20.152139 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:54:20.153403 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.138110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:60465)
0812 18:54:20.138630 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.138688 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.140065 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.140759 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.140772 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.141373 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.142141 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.142163 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.143179 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.143183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.144337 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.144347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.145002 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.145013 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.145176 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.145840 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.145858 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.147602 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:20.149929 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:20.149938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:20.149960 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:20.150307 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:20.150684 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:20.150688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:20.150690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:20.150872 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:20.151357 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:20.151368 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:20.151690 (+ 322us) client_negotiation.cc:769] Sending connection context
0812 18:54:20.151915 (+ 225us) client_negotiation.cc:240] Negotiation successful
0812 18:54:20.152337 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260812 18:54:20.155009 26788 heartbeater.cc:347] Connected to a master server at 127.16.40.254:36241
I20260812 18:54:20.155362 26788 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:20.156108 26788 heartbeater.cc:511] Master 127.16.40.254:36241 requested a full tablet report, sending...
I20260812 18:54:20.157394 26252 ts_manager.cc:194] Registered new tserver with Master: 2021b23b04094d3c8f654c8c4482e466 (127.16.40.195:33285)
I20260812 18:54:20.158102 26252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:60465
I20260812 18:54:20.169219 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
Entry for principal oryx/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab.
WARNING: no policy specified for oryx/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
Entry for principal oryx/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab.
WARNING: no policy specified for oryx/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
Entry for principal oryx/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab.
I20260812 18:54:20.275048 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26325
I20260812 18:54:20.287014 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26481
I20260812 18:54:20.300352 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26641
I20260812 18:54:20.308357 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26214
I20260812 18:54:20.317890 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36241
--webserver_interface=127.16.40.254
--webserver_port=42663
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:20.442153 26799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:20.442472 26799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:20.442515 26799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:20.446792 26799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:20.446869 26799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:20.446889 26799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:20.446907 26799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:20.446923 26799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:20.452651 26799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:20.453827 26799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:20.455310 26799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:20.464455 26809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.464455 26806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.464599 26805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.464973 26799 server_base.cc:1034] running on GCE node
I20260812 18:54:20.465626 26799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:20.466915 26799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:20.468161 26799 hybrid_clock.cc:648] HybridClock initialized: now 1786560860468149 us; error 33 us; skew 500 ppm
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560860, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:20.471458 26799 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:54:20.472644 26799 webserver.cc:460] Webserver started at http://127.16.40.254:42663/ using document root <none> and password file <none>
I20260812 18:54:20.473330 26799 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:20.473459 26799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:20.477949 26799 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:54:20.480988 26816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.482378 26799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:54:20.482594 26799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "74d5019ad7f74dc189ad97c844192606"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "57e945d06da8a96eddc61e0126306bba"
server_key_iv: "c2e524daf0eae93b5546228c01d20bf0"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.483201 26799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:20.510318 26799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:20.514179 26799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:20.514436 26799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:20.522601 26799 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:36241
I20260812 18:54:20.522698 26873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:36241 every 8 connection(s)
I20260812 18:54:20.523908 26799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:20.527107 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26799
I20260812 18:54:20.527726 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:45731
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:36241
--webserver_port=33431
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:54:20.529387 26874 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:54:20.531437 26874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Bootstrap starting.
I20260812 18:54:20.550765 26874 log.cc:824] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:20.557973 26874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:54:20.558416 26874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Bootstrap complete.
I20260812 18:54:20.562954 26874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:20.563417 26874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d5019ad7f74dc189ad97c844192606, State: Initialized, Role: FOLLOWER
I20260812 18:54:20.563983 26874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:20.564134 26874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:20.564229 26874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:20.564352 26874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:54:20.566458 26874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:20.566862 26874 leader_election.cc:302] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74d5019ad7f74dc189ad97c844192606; no voters:
I20260812 18:54:20.567291 26874 leader_election.cc:288] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 18:54:20.567709 26879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 18:54:20.568121 26879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [term 2 LEADER]: Becoming Leader. State: Replica: 74d5019ad7f74dc189ad97c844192606, State: Running, Role: LEADER
I20260812 18:54:20.568748 26874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:20.568717 26879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } }
I20260812 18:54:20.570087 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74d5019ad7f74dc189ad97c844192606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } } }
I20260812 18:54:20.570225 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:20.570107 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74d5019ad7f74dc189ad97c844192606. Latest consensus state: current_term: 2 leader_uuid: "74d5019ad7f74dc189ad97c844192606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d5019ad7f74dc189ad97c844192606" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 36241 } } }
I20260812 18:54:20.570384 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:20.570868 26888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:20.573988 26888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:20.574584 26888 catalog_manager.cc:1259] Loaded cluster ID: 91152d354f7c4ecf8d3b0b6c6bd7b770
I20260812 18:54:20.574664 26888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:20.576392 26888 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:20.577037 26888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74d5019ad7f74dc189ad97c844192606: Loaded TSK: 0
I20260812 18:54:20.578052 26888 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:54:20.665050 26876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:20.665439 26876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:20.665534 26876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:20.669992 26876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:20.670086 26876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:20.670181 26876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:20.676098 26876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.193.keytab
--principal=oryx/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:45731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=33431
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:20.677317 26876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:20.678381 26876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:20.689987 26902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.689947 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.690240 26876 server_base.cc:1034] running on GCE node
W20260812 18:54:20.690549 26907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.691076 26876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:20.691809 26876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:20.693025 26876 hybrid_clock.cc:648] HybridClock initialized: now 1786560860692997 us; error 53 us; skew 500 ppm
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560860, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:20.696411 26876 init.cc:377] Logged in from keytab as oryx/127.16.40.193@KRBTEST.COM (short username oryx)
I20260812 18:54:20.697685 26876 webserver.cc:460] Webserver started at http://127.16.40.193:33431/ using document root <none> and password file <none>
I20260812 18:54:20.698333 26876 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:20.698470 26876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:20.703341 26876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:54:20.706465 26914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.707998 26876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:20.708150 26876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "9b9bb632b0e44332aba1841cd983efdb"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "6d19989bde3371d4d979b106dbbbe151"
server_key_iv: "ce2c2710399647872a169926129e6c18"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.708856 26876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:20.721609 26876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:20.725705 26876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:20.725945 26876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:20.726743 26876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:20.727736 26876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:20.727790 26876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.727860 26876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:20.727892 26876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.739338 26876 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:45731
I20260812 18:54:20.739358 27047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:45731 every 8 connection(s)
I20260812 18:54:20.740423 26876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:20.746837 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 26876
I20260812 18:54:20.747303 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:40651
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:36241
--webserver_port=43805
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 18:54:20.754819 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.742773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:59935 (local address 127.16.40.254:36241)
0812 18:54:20.743355 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.743375 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.743762 (+ 387us) server_negotiation.cc:407] Connection header received
0812 18:54:20.744927 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.744936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.745217 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.745547 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.746718 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.747852 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.749468 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.749952 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.750143 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.754006 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:59935: BlockingRecv error: recv got EOF from 127.16.40.193:59935 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":142,"threads_started":1}
I20260812 18:54:20.754829 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.742870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:59935)
0812 18:54:20.743596 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.743638 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.744471 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.745606 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.745614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.746055 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.746515 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.746531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.747996 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.747999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.749343 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.749355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.750135 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.750139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.750328 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.750922 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.750939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.753762 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":132,"threads_started":1}
W20260812 18:54:20.755702 27048 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:36241 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.254:36241: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:20.762821 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.756318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:44401)
0812 18:54:20.756528 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.756570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.756690 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.756976 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.756981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.757198 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.757356 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.757367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.758184 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.758189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.759223 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.759239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.759752 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.759758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.759886 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.760441 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.760461 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.762661 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 18:54:20.763159 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.756374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:44401 (local address 127.16.40.254:36241)
0812 18:54:20.756503 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.756506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.756651 (+ 145us) server_negotiation.cc:407] Connection header received
0812 18:54:20.756793 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.756796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.756844 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.756923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.757521 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.758060 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.759382 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.759619 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.759711 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.762851 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:44401: BlockingRecv error: recv got EOF from 127.16.40.193:44401 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:20.769465 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.763668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:43059)
0812 18:54:20.763790 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.763806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.763892 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.764193 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.764196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.764343 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.764428 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.764433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.765514 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.765517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.766080 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.766089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.766630 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.766634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.766714 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.767251 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.767263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.769316 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 18:54:20.769774 27048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:54:20.769884 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.763751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:43059 (local address 127.16.40.254:36241)
0812 18:54:20.763876 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.763880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.763896 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:20.763995 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.763998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.764046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.764136 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.764576 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.765388 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.766229 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.766460 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.766629 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.769469 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:43059: BlockingRecv error: recv got EOF from 127.16.40.193:43059 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 18:54:20.875852 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:20.876161 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:20.876209 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:20.880349 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:20.880431 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:20.880492 27052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:20.886132 27052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.194.keytab
--principal=oryx/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:40651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=43805
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:20.887281 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:20.888375 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:20.898972 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:20.898973 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.899055 27052 server_base.cc:1034] running on GCE node
W20260812 18:54:20.898972 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:20.899688 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:20.900476 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:20.901707 27052 hybrid_clock.cc:648] HybridClock initialized: now 1786560860901688 us; error 32 us; skew 500 ppm
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560860, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:20.904937 27052 init.cc:377] Logged in from keytab as oryx/127.16.40.194@KRBTEST.COM (short username oryx)
I20260812 18:54:20.906032 27052 webserver.cc:460] Webserver started at http://127.16.40.194:43805/ using document root <none> and password file <none>
I20260812 18:54:20.906736 27052 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:20.906859 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:20.911151 27052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 18:54:20.913710 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.915162 27052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:20.915313 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "c8dcec90a1bf4d53a51b8120f34a8ffd"
format_stamp: "Formatted at 2026-08-12 18:54:19 on dist-test-slave-rnrw"
server_key: "9c2141be2170d9b85d6c6ccfa52b961f"
server_key_iv: "bea4c55a09b4f99fe1b658712d45646f"
server_key_version: "encryptionkey@0"
I20260812 18:54:20.915910 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:20.943802 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:20.947510 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:20.947767 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:20.948436 27052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:20.949368 27052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:20.949443 27052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.949517 27052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:20.949565 27052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:20.959260 27052 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:40651
I20260812 18:54:20.959288 27198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:40651 every 8 connection(s)
I20260812 18:54:20.960397 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:20.966791 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27052
I20260812 18:54:20.967307 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:33285
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:36241
--webserver_port=40861
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 18:54:20.974651 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.962813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:50519 (local address 127.16.40.254:36241)
0812 18:54:20.962984 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.962989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.964232 (+ 1243us) server_negotiation.cc:407] Connection header received
0812 18:54:20.965242 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.965245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.965301 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.965386 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.966908 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.967628 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.969632 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.970089 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.970243 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.974471 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:50519: BlockingRecv error: recv got EOF from 127.16.40.194:50519 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 18:54:20.975186 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.963236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:50519)
0812 18:54:20.964059 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.964103 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.965020 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.965657 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.965665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.966056 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.966697 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.966722 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.967769 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.967776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.969387 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.969399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.970290 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.970293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.970516 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.971216 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.971235 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.974235 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":118,"threads_started":1}
W20260812 18:54:20.975883 27199 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:36241 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.254:36241: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:20.982549 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.976400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:38875)
0812 18:54:20.976713 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.976725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.976798 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.977118 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.977120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.977264 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.977361 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.977366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.978490 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.978493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.979046 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.979054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.979735 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.979738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.979815 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.980434 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.980445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.982396 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239}
W20260812 18:54:20.982689 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.976584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:38875 (local address 127.16.40.254:36241)
0812 18:54:20.976735 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.976741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.976766 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:54:20.976910 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.976916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.976982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.977096 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.977481 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.978369 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.979168 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.979608 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.979750 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.982567 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:38875: BlockingRecv error: recv got EOF from 127.16.40.194:38875 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:20 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:20.988756 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:20.983242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:51657)
0812 18:54:20.983348 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:20.983364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:20.983445 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:20.983689 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:20.983691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:20.983820 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:20.983897 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.983901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.984926 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.984929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.985442 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:20.985450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.985992 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.985994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:20.986074 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.986582 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:20.986593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:20.988646 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 18:54:20.988924 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:20.983294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:51657 (local address 127.16.40.254:36241)
0812 18:54:20.983396 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:20.983400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:20.983414 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:20.983533 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:20.983535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:20.983579 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:20.983662 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:20.983997 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.984820 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.985549 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:20.985884 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:20.985984 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:20.988803 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:51657: BlockingRecv error: recv got EOF from 127.16.40.194:51657 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 18:54:20.989394 27199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:54:21.090662 27202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:21.091033 27202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:21.091094 27202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:21.095733 27202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:21.095801 27202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:21.095865 27202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:21.101532 27202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/krb5kdc/oryx_127.16.40.195.keytab
--principal=oryx/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:33285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=40861
--tserver_master_addrs=127.16.40.254:36241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:21.102695 27202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:21.103751 27202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:21.114566 27209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:21.114624 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:21.114565 27208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:21.115164 27202 server_base.cc:1034] running on GCE node
I20260812 18:54:21.115646 27202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:21.116415 27202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:21.117664 27202 hybrid_clock.cc:648] HybridClock initialized: now 1786560861117638 us; error 35 us; skew 500 ppm
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560861, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:21.120960 27202 init.cc:377] Logged in from keytab as oryx/127.16.40.195@KRBTEST.COM (short username oryx)
I20260812 18:54:21.121980 27202 webserver.cc:460] Webserver started at http://127.16.40.195:40861/ using document root <none> and password file <none>
I20260812 18:54:21.122617 27202 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:21.122716 27202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:21.127204 27202 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:54:21.129547 27219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:21.130908 27202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 18:54:21.131049 27202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "2021b23b04094d3c8f654c8c4482e466"
format_stamp: "Formatted at 2026-08-12 18:54:20 on dist-test-slave-rnrw"
server_key: "29fc8399890440c0f1c78e61bfaa40b2"
server_key_iv: "9bcd5e06ee1d6115ce810f68aa7ff841"
server_key_version: "encryptionkey@0"
I20260812 18:54:21.131672 27202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:21.159370 27202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:21.164239 27202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:21.164427 27202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:21.165210 27202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:21.166100 27202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:21.166152 27202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:21.166195 27202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:21.166224 27202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:21.178977 27202 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:33285
I20260812 18:54:21.179111 27354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:33285 every 8 connection(s)
I20260812 18:54:21.180519 27202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:21.187973 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27202
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 18:54:21.201602 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:21.183625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:35241 (local address 127.16.40.254:36241)
0812 18:54:21.183812 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.183817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.185478 (+ 1661us) server_negotiation.cc:407] Connection header received
0812 18:54:21.186824 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.186834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.186917 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.187042 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.190720 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.191813 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.194377 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.194990 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.195200 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.201370 (+ 6170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:35241: BlockingRecv error: recv got EOF from 127.16.40.195:35241 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:54:21.202533 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.184114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:35241)
0812 18:54:21.185195 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:21.185276 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:21.186350 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:21.187562 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:21.187592 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:21.188737 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:21.190357 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.190389 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.192030 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.192040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.194201 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.194220 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.195270 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.195275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.195609 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.196957 (+ 1348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:21.197021 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:21.201101 (+ 4080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":156,"threads_started":1}
W20260812 18:54:21.203624 27355 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:36241 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.40.254:36241: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
I20260812 18:54:21.206910 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.190318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.16.40.254:36241)
0812 18:54:21.190754 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.190763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.190785 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:54:21.190839 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.190846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.190909 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.191040 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.192646 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.193763 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.195604 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.196112 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.196417 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.198784 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:21.198803 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:21.198823 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:21.198891 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:21.202289 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:21.203222 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:21.203232 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:21.203239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:21.203829 (+ 590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:21.204686 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:21.204702 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:21.204706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:21.205640 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:21.206019 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:21.206444 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 18:54:21.206735 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:21.213207 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.204733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:39941)
0812 18:54:21.204955 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:21.204971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:21.205125 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:21.205575 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:21.205580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:21.205954 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:21.206159 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.206165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.207515 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.207526 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.208306 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.208317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.209033 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.209038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.209185 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.209958 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:21.209974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:21.213048 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260812 18:54:21.213568 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:21.204871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:39941 (local address 127.16.40.254:36241)
0812 18:54:21.205017 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.205021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.205034 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:54:21.205302 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.205305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.205358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.205487 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.206330 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.207298 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.208489 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.208848 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.209018 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.213291 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:39941: BlockingRecv error: recv got EOF from 127.16.40.195:39941 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:21.221798 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.214127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:59749)
0812 18:54:21.214330 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:21.214367 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:21.214470 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:21.214714 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:21.214716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:21.214861 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:21.214973 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.214978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.215873 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.215884 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.216820 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.216842 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.217479 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.217482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.217629 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.218620 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:21.218640 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:21.221643 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260812 18:54:21.222304 27355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:54:21.222425 27050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:21.214215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:59749 (local address 127.16.40.254:36241)
0812 18:54:21.214459 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.214462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.214477 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:21.214554 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.214556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.214600 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.214695 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.215123 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.215714 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.217019 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.217348 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.217467 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.222217 (+ 4750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:59749: BlockingRecv error: recv got EOF from 127.16.40.195:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:21.776850 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.770589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:51639)
0812 18:54:21.770875 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:21.770892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:21.771015 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:21.771339 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:21.771342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:21.771562 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:21.771679 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.771685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.772712 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.772716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.773473 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.773485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.773982 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.773985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.774073 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.774566 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:21.774579 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:21.776716 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
W20260812 18:54:21.777149 27362 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:21.770685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51639 (local address 127.16.40.254:36241)
0812 18:54:21.770933 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.770936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.770951 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:21.771116 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.771119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.771166 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.771255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.771820 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.772527 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.773614 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.773852 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.773977 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.776976 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:51639: BlockingRecv error: recv got EOF from 127.16.40.193:51639 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:21 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:21.996685 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:21.990231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:40371)
0812 18:54:21.990524 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:21.990538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:21.990667 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:21.990989 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:21.990991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:21.991277 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:21.991419 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.991426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.992465 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.992468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.993202 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:21.993213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.993724 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.993727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:21.993787 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.994349 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:21.994364 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:21.996438 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260812 18:54:21.996758 27362 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:21.990337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:40371 (local address 127.16.40.254:36241)
0812 18:54:21.990487 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:21.990490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:21.990621 (+ 131us) server_negotiation.cc:407] Connection header received
0812 18:54:21.990773 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:21.990775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:21.990849 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:21.990955 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:21.991553 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.992361 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.993342 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:21.993611 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:21.993703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:21.996664 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:40371: BlockingRecv error: recv got EOF from 127.16.40.194:40371 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:22.229147 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:22.223081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:54133)
0812 18:54:22.223405 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:22.223421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:22.223544 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:22.223827 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:22.223830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:22.224046 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:22.224179 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.224186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.225144 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.225148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:22.225789 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.225799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.226313 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.226316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:22.226395 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:22.226942 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:22.226955 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:22.229006 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260812 18:54:22.229362 27362 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:22.223245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:54133 (local address 127.16.40.254:36241)
0812 18:54:22.223439 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:22.223443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:22.223483 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:54:22.223641 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:22.223643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:22.223693 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:22.223772 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:22.224312 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.225020 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.225946 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.226198 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.226291 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:22.229176 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:54133: BlockingRecv error: recv got EOF from 127.16.40.195:54133 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:22.784827 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:22.777902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:55079)
0812 18:54:22.778228 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:22.778244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:22.778361 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:22.778682 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:22.778685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:22.778896 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:22.778995 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.779001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.779789 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.779794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:22.780398 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.780407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.781148 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.781154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:22.781276 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:22.782010 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:22.782026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:22.784610 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
W20260812 18:54:22.785158 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:22.778062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:55079 (local address 127.16.40.254:36241)
0812 18:54:22.778342 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:22.778348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:22.778362 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:22.778486 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:22.778492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:22.778544 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:22.778653 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:22.779123 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.779673 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.780526 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.780947 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.781065 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:22.785016 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:55079: BlockingRecv error: recv got EOF from 127.16.40.193:55079 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:22 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:23.003357 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:22.997550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:56787)
0812 18:54:22.997836 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:22.997850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:22.997951 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:22.998202 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:22.998204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:22.998391 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:22.998551 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.998557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.999287 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.999290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:22.999880 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:22.999889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.000312 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.000315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:23.000383 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.001101 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:23.001114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:23.003156 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
W20260812 18:54:23.003388 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:22.997718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:56787 (local address 127.16.40.254:36241)
0812 18:54:22.997880 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:22.997883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:22.997901 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:54:22.998041 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:22.998044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:22.998088 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:22.998160 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:22.998659 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:22.999182 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:22.999986 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.000197 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.000272 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.003286 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:56787: BlockingRecv error: recv got EOF from 127.16.40.194:56787 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 18:54:23 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:23 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:23.235863 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:23.229943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:43561)
0812 18:54:23.230280 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:23.230297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:23.230449 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:23.230788 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:23.230791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:23.231022 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:23.231120 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:23.231125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.231896 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.231902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:23.232455 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:23.232463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.232966 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.232970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:23.233062 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.233607 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:23.233619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:23.235653 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
W20260812 18:54:23.236035 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:23.230139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43561 (local address 127.16.40.254:36241)
0812 18:54:23.230281 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:23.230287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:23.230388 (+ 101us) server_negotiation.cc:407] Connection header received
0812 18:54:23.230626 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:23.230629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:23.230677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:23.230765 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:23.231247 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.231781 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.232624 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.232847 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.232961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.235875 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:43561: BlockingRecv error: recv got EOF from 127.16.40.195:43561 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 18:54:23 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:23 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:23.793394 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:23.786519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:39105)
0812 18:54:23.786872 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:23.786887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:23.787020 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:23.787352 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:23.787355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:23.787648 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:23.787799 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:23.787808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.788984 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.788989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:23.789722 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:23.789732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.790394 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.790397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:23.790460 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.791007 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:23.791021 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:23.793276 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
W20260812 18:54:23.793632 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:23.786571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:39105 (local address 127.16.40.254:36241)
0812 18:54:23.786965 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:23.786972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:23.786992 (+ 20us) server_negotiation.cc:407] Connection header received
0812 18:54:23.787139 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:23.787144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:23.787206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:23.787300 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:23.787922 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.788813 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.789854 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:23.790267 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:23.790417 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:23.793489 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:39105: BlockingRecv error: recv got EOF from 127.16.40.193:39105 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:24.009943 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:24.004290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:57365)
0812 18:54:24.004649 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:24.004663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:24.004770 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:24.005017 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:24.005019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:24.005206 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:24.005312 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.005317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.006127 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.006130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.006713 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.006727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.007181 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.007183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.007248 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.007777 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:24.007790 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:24.009745 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260812 18:54:24.009982 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:24.004402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:57365 (local address 127.16.40.254:36241)
0812 18:54:24.004597 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:24.004600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:24.004730 (+ 130us) server_negotiation.cc:407] Connection header received
0812 18:54:24.004852 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:24.004854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:24.004906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:24.004978 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:24.005452 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.006018 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.006832 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.007070 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.007153 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.009882 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:57365: BlockingRecv error: recv got EOF from 127.16.40.194:57365 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:24.242738 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:24.236757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:35765)
0812 18:54:24.237085 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:24.237102 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:24.237226 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:24.237564 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:24.237567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:24.237788 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:24.237889 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.237894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.238729 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.238736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.239320 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.239329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.239806 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.239810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.239899 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.240602 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:24.240616 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:24.242600 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
W20260812 18:54:24.243037 27370 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:24.236978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:35765 (local address 127.16.40.254:36241)
0812 18:54:24.237143 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:24.237150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:24.237168 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:54:24.237336 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:24.237341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:24.237401 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:24.237531 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:24.238026 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.238610 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.239455 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.239684 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.239798 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.242883 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:35765: BlockingRecv error: recv got EOF from 127.16.40.195:35765 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:24 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:24.803754 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:24.794269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:51579)
0812 18:54:24.794615 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:24.794631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:24.794731 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:24.794964 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:24.794966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:24.795153 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:24.795251 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.795256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.795999 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.796001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.796627 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:24.796637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.797107 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.797110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:24.797176 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.801303 (+ 4127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:24.801318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:24.803627 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
W20260812 18:54:24.803916 27375 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:24.794395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51579 (local address 127.16.40.254:36241)
0812 18:54:24.794680 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:24.794683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:24.794698 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:24.794804 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:24.794806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:24.794850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:24.794929 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:24.795355 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.795896 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.796741 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:24.796972 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:24.797088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:24.803777 (+ 6689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:51579: BlockingRecv error: recv got EOF from 127.16.40.193:51579 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:25.016479 27377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:25.010811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:53425)
0812 18:54:25.011084 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:25.011098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:25.011194 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:25.011430 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:25.011432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:25.011621 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:25.011720 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.011725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.012473 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.012508 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.013249 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.013261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.013711 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.013713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.013834 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.014365 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:25.014378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:25.016299 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
W20260812 18:54:25.016584 27375 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:25.010869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:53425 (local address 127.16.40.254:36241)
0812 18:54:25.011026 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:25.011029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:25.011165 (+ 136us) server_negotiation.cc:407] Connection header received
0812 18:54:25.011273 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:25.011275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:25.011320 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:25.011394 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:25.011828 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.012371 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.013368 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.013592 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.013679 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.016442 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:53425: BlockingRecv error: recv got EOF from 127.16.40.194:53425 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:25.249514 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:25.243610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:49031)
0812 18:54:25.243963 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:25.243981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:25.244156 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:25.244449 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:25.244452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:25.244720 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:25.244822 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.244829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.245603 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.245609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.246166 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.246174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.246654 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.246658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.246745 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.247342 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:25.247355 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:25.249327 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
W20260812 18:54:25.249607 27375 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:25.243769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:49031 (local address 127.16.40.254:36241)
0812 18:54:25.243948 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:25.243954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:25.244091 (+ 137us) server_negotiation.cc:407] Connection header received
0812 18:54:25.244265 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:25.244268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:25.244327 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:25.244421 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:25.244949 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.245479 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.246291 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.246531 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.246648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.249476 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:49031: BlockingRecv error: recv got EOF from 127.16.40.195:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:25 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:25.810600 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:25.804668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:44007)
0812 18:54:25.804943 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:25.804963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:25.805094 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:25.805354 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:25.805357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:25.805565 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:25.805661 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.805666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.806549 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.806553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.807148 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:25.807157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.807731 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.807734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:25.807804 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.808397 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:25.808409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:25.810467 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
W20260812 18:54:25.810985 27380 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:25.804798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:44007 (local address 127.16.40.254:36241)
0812 18:54:25.805092 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:25.805096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:25.805111 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:25.805157 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:25.805159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:25.805199 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:25.805284 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:25.805805 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.806354 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.807289 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:25.807577 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:25.807687 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:25.810829 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:44007: BlockingRecv error: recv got EOF from 127.16.40.193:44007 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:26.023442 27381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:26.017628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:51625)
0812 18:54:26.018012 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:26.018030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:26.018140 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:26.018417 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:26.018420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:26.018623 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:26.018726 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.018732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.019596 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.019600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.020227 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.020238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.020771 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.020774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.020845 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.021386 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:26.021398 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:26.023320 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
W20260812 18:54:26.023645 27380 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:26.017793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:51625 (local address 127.16.40.254:36241)
0812 18:54:26.017948 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:26.017954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:26.018113 (+ 159us) server_negotiation.cc:407] Connection header received
0812 18:54:26.018217 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:26.018220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:26.018264 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:26.018336 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:26.018838 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.019432 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.020336 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.020635 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.020721 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.023471 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:51625: BlockingRecv error: recv got EOF from 127.16.40.194:51625 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:26.256356 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:26.250633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:58259)
0812 18:54:26.250948 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:26.250961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:26.251089 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:26.251365 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:26.251367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:26.251598 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:26.251694 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.251700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.252485 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.252489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.253107 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.253117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.253599 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.253603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.253690 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.254274 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:26.254287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:26.256178 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
W20260812 18:54:26.256469 27380 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:26.250624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:58259 (local address 127.16.40.254:36241)
0812 18:54:26.250750 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:26.250754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:26.251039 (+ 285us) server_negotiation.cc:407] Connection header received
0812 18:54:26.251188 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:26.251191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:26.251249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:26.251335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:26.251819 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.252355 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.253235 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.253478 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.253591 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.256336 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:58259: BlockingRecv error: recv got EOF from 127.16.40.195:58259 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:26 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:26.817384 27383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:26.811478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:35321)
0812 18:54:26.811831 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:26.811845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:26.811937 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:26.812285 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:26.812287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:26.812593 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:26.812728 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.812735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.813583 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.813586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.814268 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:26.814279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.814724 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.814726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:26.814789 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.815298 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:26.815310 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:26.817260 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
W20260812 18:54:26.817677 27384 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:26.811514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:35321 (local address 127.16.40.254:36241)
0812 18:54:26.811878 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:26.811886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:26.811901 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:26.812104 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:26.812107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:26.812158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:26.812247 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:26.812885 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.813449 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.814378 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:26.814615 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:26.814699 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:26.817505 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:35321: BlockingRecv error: recv got EOF from 127.16.40.193:35321 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":144,"threads_started":1}
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:27.030438 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:27.024402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:51541)
0812 18:54:27.024737 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:27.024751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:27.024854 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:27.025125 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:27.025127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:27.025317 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:27.025418 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.025423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.026252 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.026255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.027051 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.027064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.027682 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.027689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.027786 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.028301 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:27.028313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:27.030304 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
W20260812 18:54:27.030575 27384 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:27.024497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:51541 (local address 127.16.40.254:36241)
0812 18:54:27.024747 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:27.024751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:27.024823 (+ 72us) server_negotiation.cc:407] Connection header received
0812 18:54:27.024941 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:27.024944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:27.024990 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:27.025090 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:27.025536 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.026122 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.027221 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.027550 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.027668 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.030463 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:51541: BlockingRecv error: recv got EOF from 127.16.40.194:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:27.263964 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:27.257483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:49379)
0812 18:54:27.257803 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:27.257818 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:27.257964 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:27.258450 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:27.258454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:27.258717 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:27.258825 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.258831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.259697 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.259700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.260267 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.260276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.260911 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.260915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.260992 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.261606 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:27.261620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:27.263815 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":117,"threads_started":1}
W20260812 18:54:27.264184 27384 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:27.257650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:49379 (local address 127.16.40.254:36241)
0812 18:54:27.257797 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:27.257808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:27.257915 (+ 107us) server_negotiation.cc:407] Connection header received
0812 18:54:27.258186 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:27.258191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:27.258261 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:27.258372 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:27.258924 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.259571 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.260425 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.260758 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.260862 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.264036 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:49379: BlockingRecv error: recv got EOF from 127.16.40.195:49379 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:27 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:27.824774 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:27.818254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:34037)
0812 18:54:27.818618 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:27.818633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:27.818731 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:27.819105 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:27.819110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:27.819388 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:27.819494 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.819500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.820714 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.820720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.821323 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:27.821333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.821824 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.821828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:27.821917 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.822461 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:27.822474 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:27.824649 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":81,"threads_started":1}
W20260812 18:54:27.824906 27389 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:27.818393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:34037 (local address 127.16.40.254:36241)
0812 18:54:27.818707 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:27.818710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:27.818727 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:54:27.818856 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:27.818859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:27.818918 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:27.819026 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:27.819663 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.820511 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.821467 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:27.821699 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:27.821815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:27.824767 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:34037: BlockingRecv error: recv got EOF from 127.16.40.193:34037 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":107,"threads_started":1}
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:28.038002 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:28.031478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:33771)
0812 18:54:28.031847 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:28.031861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:28.031968 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:28.032310 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:28.032313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:28.032517 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:28.032674 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.032680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.033451 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.033454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.034047 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.034056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.034611 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.034615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.034706 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.035345 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:28.035358 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:28.037831 (+ 2473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":137,"threads_started":1}
W20260812 18:54:28.038334 27389 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:28.031613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:33771 (local address 127.16.40.254:36241)
0812 18:54:28.031767 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:28.031771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:28.031940 (+ 169us) server_negotiation.cc:407] Connection header received
0812 18:54:28.032113 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:28.032116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:28.032174 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:28.032255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:28.032800 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.033340 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.034200 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.034482 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.034599 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.038051 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:33771: BlockingRecv error: recv got EOF from 127.16.40.194:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:28.271377 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:28.265064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:58887)
0812 18:54:28.265408 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:28.265423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:28.265594 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:28.265887 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:28.265890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:28.266090 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:28.266189 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.266195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.267083 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.267087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.267778 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.267790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.268432 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.268435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.268538 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.269240 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:28.269252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:28.271254 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
W20260812 18:54:28.271600 27389 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:28.265254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:58887 (local address 127.16.40.254:36241)
0812 18:54:28.265445 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:28.265451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:28.265554 (+ 103us) server_negotiation.cc:407] Connection header received
0812 18:54:28.265681 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:28.265684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:28.265735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:28.265810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:28.266307 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.266889 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.267893 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.268238 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.268384 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.271447 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:58887: BlockingRecv error: recv got EOF from 127.16.40.195:58887 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:28 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:28.831991 27392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:28.825765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:36677)
0812 18:54:28.826102 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:28.826117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:28.826220 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:28.826520 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:28.826522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:28.826708 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:28.826807 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.826812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.827621 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.827625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.828207 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:28.828216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.828727 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.828731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:28.828810 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.829455 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:28.829469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:28.831754 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":129,"threads_started":1}
W20260812 18:54:28.832127 27393 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:28.825985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:36677 (local address 127.16.40.254:36241)
0812 18:54:28.826237 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:28.826241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:28.826256 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:54:28.826297 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:28.826299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:28.826342 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:28.826485 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:28.826911 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.827485 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.828318 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:28.828619 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:28.828698 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:28.831977 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:36677: BlockingRecv error: recv got EOF from 127.16.40.193:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:29.045351 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:29.038994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:55487)
0812 18:54:29.039312 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:29.039328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:29.039542 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:29.039843 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:29.039846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:29.040078 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:29.040180 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.040187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.041194 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.041199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.041770 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.041779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.042291 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.042295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.042385 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.042947 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:29.042960 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:29.045152 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
W20260812 18:54:29.045411 27393 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:29.039095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:55487 (local address 127.16.40.254:36241)
0812 18:54:29.039286 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:29.039290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:29.039429 (+ 139us) server_negotiation.cc:407] Connection header received
0812 18:54:29.039660 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:29.039663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:29.039722 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:29.039818 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:29.040315 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.041072 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.041918 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.042160 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.042273 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.045298 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:55487: BlockingRecv error: recv got EOF from 127.16.40.194:55487 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:29.277977 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:29.272249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:55077)
0812 18:54:29.272604 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:29.272623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:29.272782 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:29.273074 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:29.273077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:29.273290 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:29.273383 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.273389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.274156 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.274161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.274721 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.274729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.275198 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.275202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.275291 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.275814 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:29.275826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:29.277785 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
W20260812 18:54:29.278033 27393 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:29.272325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:55077 (local address 127.16.40.254:36241)
0812 18:54:29.272605 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:29.272611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:29.272719 (+ 108us) server_negotiation.cc:407] Connection header received
0812 18:54:29.272890 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:29.272893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:29.272954 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:29.273047 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:29.273510 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.274042 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.274854 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.275077 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.275191 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.277920 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:55077: BlockingRecv error: recv got EOF from 127.16.40.195:55077 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:29 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:29.847908 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:29.833160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:51491)
0812 18:54:29.833476 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:29.833493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:29.833616 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:29.833939 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:29.833941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:29.834175 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:29.834282 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.834288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.835300 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.835306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.836145 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:29.836157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.836719 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.836722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:29.836793 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.845172 (+ 8379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:29.845194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:29.847695 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":134,"threads_started":1}
W20260812 18:54:29.848029 27398 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:29.833281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51491 (local address 127.16.40.254:36241)
0812 18:54:29.833547 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:29.833550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:29.833566 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:29.833711 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:29.833714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:29.833760 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:29.833847 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:29.834397 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.835145 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.836261 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:29.836568 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:29.836710 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:29.847857 (+ 11147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:51491: BlockingRecv error: recv got EOF from 127.16.40.193:51491 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:30.053231 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:30.046613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:46093)
0812 18:54:30.047007 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:30.047021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:30.047179 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:30.047546 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:30.047548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:30.047771 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:30.047882 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.047888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.048753 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.048758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.049365 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.049374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.049882 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.049886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.049982 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.050648 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:30.050661 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:30.052994 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":162,"threads_started":1}
W20260812 18:54:30.053249 27398 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:30.046624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:46093 (local address 127.16.40.254:36241)
0812 18:54:30.046839 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:30.046845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:30.047115 (+ 270us) server_negotiation.cc:407] Connection header received
0812 18:54:30.047290 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:30.047293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:30.047352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:30.047522 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:30.048019 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.048635 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.049501 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.049751 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.049870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.053137 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:46093: BlockingRecv error: recv got EOF from 127.16.40.194:46093 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:30.285992 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:30.279037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:43679)
0812 18:54:30.279480 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:30.279496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:30.279611 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:30.279966 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:30.279969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:30.280177 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:30.280280 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.280286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.281289 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.281292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.281906 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.281916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.282388 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.282391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.282505 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.283134 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:30.283146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:30.285772 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":237,"threads_started":1}
W20260812 18:54:30.286276 27398 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:30.279162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:43679 (local address 127.16.40.254:36241)
0812 18:54:30.279309 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:30.279313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:30.279605 (+ 292us) server_negotiation.cc:407] Connection header received
0812 18:54:30.279714 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:30.279716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:30.279806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:30.279910 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:30.280483 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.281161 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.282026 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.282276 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.282366 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.286108 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:43679: BlockingRecv error: recv got EOF from 127.16.40.195:43679 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:30 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:30.854951 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:30.848990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:39253)
0812 18:54:30.849327 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:30.849341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:30.849429 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:30.849726 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:30.849728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:30.849910 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:30.850007 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.850013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.850829 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.850831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.851381 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:30.851388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.851884 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.851887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:30.851950 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.852460 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:30.852491 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:30.854832 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":134,"threads_started":1}
W20260812 18:54:30.855145 27403 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:30.849189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:39253 (local address 127.16.40.254:36241)
0812 18:54:30.849416 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:30.849420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:30.849434 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:30.849519 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:30.849521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:30.849605 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:30.849687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:30.850124 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.850724 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.851517 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:30.851756 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:30.851852 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:30.854973 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:39253: BlockingRecv error: recv got EOF from 127.16.40.193:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:31.059696 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:31.054095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:54207)
0812 18:54:31.054406 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:31.054419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:31.054521 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:31.054813 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:31.054815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:31.054998 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:31.055092 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.055097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.055999 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.056002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.056633 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.056649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.057153 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.057156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.057218 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.057736 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:31.057748 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:31.059570 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
W20260812 18:54:31.059947 27403 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:31.054191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:54207 (local address 127.16.40.254:36241)
0812 18:54:31.054377 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:31.054381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:31.054483 (+ 102us) server_negotiation.cc:407] Connection header received
0812 18:54:31.054631 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:31.054634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:31.054685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:31.054772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:31.055243 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.055891 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.056779 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.057027 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.057115 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.059792 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:54207: BlockingRecv error: recv got EOF from 127.16.40.194:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:31.293283 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:31.287109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:38071)
0812 18:54:31.287498 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:31.287567 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:31.287715 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:31.288096 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:31.288099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:31.288293 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:31.288436 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.288447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.289254 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.289257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.289839 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.289848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.290340 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.290344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.290414 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.291034 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:31.291046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:31.293108 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
W20260812 18:54:31.293557 27403 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:31.287286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:38071 (local address 127.16.40.254:36241)
0812 18:54:31.287500 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:31.287506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:31.287673 (+ 167us) server_negotiation.cc:407] Connection header received
0812 18:54:31.287842 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:31.287846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:31.287909 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:31.288029 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:31.288619 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.289154 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.289959 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.290194 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.290281 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.293289 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:38071: BlockingRecv error: recv got EOF from 127.16.40.195:38071 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:31 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:31.862191 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:31.855933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:56981)
0812 18:54:31.856282 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:31.856296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:31.856394 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:31.856825 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:31.856827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:31.857052 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:31.857158 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.857164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.858003 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.858006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.858579 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:31.858589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.859057 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.859059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:31.859122 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.859690 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:31.859702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:31.862070 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":146,"threads_started":1}
W20260812 18:54:31.862382 27408 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:31.856089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:56981 (local address 127.16.40.254:36241)
0812 18:54:31.856342 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:31.856346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:31.856362 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:31.856627 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:31.856630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:31.856683 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:31.856780 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:31.857278 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.857858 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.858715 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:31.858943 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:31.859042 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:31.862207 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:56981: BlockingRecv error: recv got EOF from 127.16.40.193:56981 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:32.066627 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:32.060589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:48293)
0812 18:54:32.060872 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:32.060886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:32.060990 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:32.061242 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:32.061244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:32.061439 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:32.061577 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.061587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.062365 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.062368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.062925 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.062936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.063421 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.063473 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.063538 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.064115 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:32.064128 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:32.066482 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
W20260812 18:54:32.066740 27408 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:32.060695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:48293 (local address 127.16.40.254:36241)
0812 18:54:32.060857 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:32.060860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:32.060987 (+ 127us) server_negotiation.cc:407] Connection header received
0812 18:54:32.061081 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:32.061083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:32.061133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:32.061201 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:32.061707 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.062241 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.063068 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.063304 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.063450 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.066635 (+ 3185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:48293: BlockingRecv error: recv got EOF from 127.16.40.194:48293 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:32.300119 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:32.294185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:42363)
0812 18:54:32.294487 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:32.294506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:32.294608 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:32.294870 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:32.294872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:32.295063 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:32.295166 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.295172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.295978 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.295982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.296714 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.296726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.297233 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.297236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.297307 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.297833 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:32.297846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:32.299974 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
W20260812 18:54:32.300345 27408 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:32.294232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:42363 (local address 127.16.40.254:36241)
0812 18:54:32.294421 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:32.294425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:32.294580 (+ 155us) server_negotiation.cc:407] Connection header received
0812 18:54:32.294700 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:32.294703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:32.294753 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:32.294829 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:32.295292 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.295840 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.296836 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.297092 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.297182 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.300196 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:42363: BlockingRecv error: recv got EOF from 127.16.40.195:42363 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:32 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:32.869617 27411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:32.863184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:49453)
0812 18:54:32.863661 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:32.863681 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:32.863828 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:32.864116 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:32.864118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:32.864329 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:32.864433 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.864438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.865225 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.865227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.865796 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:32.865805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.866288 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.866291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:32.866355 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.866875 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:32.866887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:32.869473 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":209,"threads_started":1}
W20260812 18:54:32.869948 27412 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:32.863279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:49453 (local address 127.16.40.254:36241)
0812 18:54:32.863653 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:32.863657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:32.863850 (+ 193us) server_negotiation.cc:407] Connection header received
0812 18:54:32.863907 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:32.863909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:32.863959 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:32.864060 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:32.864570 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.865118 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.865922 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:32.866175 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:32.866264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:32.869682 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:49453: BlockingRecv error: recv got EOF from 127.16.40.193:49453 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":178,"threads_started":1}
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:33.074393 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:33.067796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:50793)
0812 18:54:33.068088 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:33.068106 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:33.068217 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:33.068624 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:33.068626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:33.068823 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:33.068935 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.068942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.069988 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.069995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.070701 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.070714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.071322 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.071329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.071421 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.072051 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:33.072071 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:33.074243 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":132,"threads_started":1}
W20260812 18:54:33.074642 27412 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:33.067939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:50793 (local address 127.16.40.254:36241)
0812 18:54:33.068089 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:33.068093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:33.068186 (+ 93us) server_negotiation.cc:407] Connection header received
0812 18:54:33.068343 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:33.068346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:33.068448 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:33.068533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:33.069074 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.069782 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.070887 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.071151 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.071254 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.074509 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:50793: BlockingRecv error: recv got EOF from 127.16.40.194:50793 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:33.306856 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:33.300982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:33401)
0812 18:54:33.301380 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:33.301394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:33.301492 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:33.301731 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:33.301733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:33.301921 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:33.302039 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.302045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.302886 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.302889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.303475 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.303485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.303944 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.303946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.304012 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.304735 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:33.304747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:33.306734 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
W20260812 18:54:33.307050 27412 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:33.301079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:33401 (local address 127.16.40.254:36241)
0812 18:54:33.301217 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:33.301223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:33.301454 (+ 231us) server_negotiation.cc:407] Connection header received
0812 18:54:33.301575 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:33.301578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:33.301629 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:33.301691 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:33.302157 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.302744 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.303610 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.303834 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.303916 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.306906 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:33401: BlockingRecv error: recv got EOF from 127.16.40.195:33401 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:33 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:33.876394 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:33.870648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:51603)
0812 18:54:33.870987 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:33.871006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:33.871139 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:33.871433 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:33.871436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:33.871650 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:33.871744 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.871750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.872599 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.872605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.873187 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:33.873196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.873685 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.873688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:33.873775 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.874374 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:33.874386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:33.876254 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
W20260812 18:54:33.876972 27417 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:33.870733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:51603 (local address 127.16.40.254:36241)
0812 18:54:33.871012 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:33.871017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:33.871072 (+ 55us) server_negotiation.cc:407] Connection header received
0812 18:54:33.871249 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:33.871252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:33.871309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:33.871409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:33.871873 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.872420 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.873317 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:33.873562 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:33.873679 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:33.876762 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:51603: BlockingRecv error: recv got EOF from 127.16.40.193:51603 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:34.081455 27418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:34.075385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:38957)
0812 18:54:34.075700 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:34.075719 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:34.075840 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:34.076126 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:34.076129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:34.076375 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:34.076484 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.076490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.077372 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.077375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.078001 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.078013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.078501 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.078543 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.078615 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.079181 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:34.079194 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:34.081325 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
W20260812 18:54:34.081696 27417 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:34.075530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:38957 (local address 127.16.40.254:36241)
0812 18:54:34.075734 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:34.075741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:34.075781 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:54:34.075944 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:34.075951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:34.075998 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:34.076076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:34.076651 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.077210 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.078138 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.078373 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.078458 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.081536 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:38957: BlockingRecv error: recv got EOF from 127.16.40.194:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:34.314121 27419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:34.307953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:44429)
0812 18:54:34.308253 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:34.308268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:34.308415 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:34.308764 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:34.308767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:34.308979 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:34.309077 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.309084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.309879 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.309884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.310453 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.310461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.311005 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.311009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.311103 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.311667 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:34.311679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:34.313930 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
W20260812 18:54:34.314198 27417 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:34.308027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:44429 (local address 127.16.40.254:36241)
0812 18:54:34.308186 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:34.308190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:34.308352 (+ 162us) server_negotiation.cc:407] Connection header received
0812 18:54:34.308527 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:34.308530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:34.308610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:34.308702 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:34.309206 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.309755 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.310582 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.310829 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.310962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.314082 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:44429: BlockingRecv error: recv got EOF from 127.16.40.195:44429 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:34 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:34.889706 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:34.877551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:59259)
0812 18:54:34.877901 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:34.877915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:34.878016 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:34.878286 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:34.878289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:34.878488 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:34.878594 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.878600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.879362 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.879364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.879949 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:34.879959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.880472 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.880475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:34.880583 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.887035 (+ 6452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:34.887057 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:34.889552 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
W20260812 18:54:34.890130 27422 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:34.877673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:59259 (local address 127.16.40.254:36241)
0812 18:54:34.878006 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:34.878010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:34.878026 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:34.878098 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:34.878101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:34.878148 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:34.878243 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:34.878721 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.879258 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.880108 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:34.880353 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:34.880442 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:34.889981 (+ 9539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:59259: BlockingRecv error: recv got EOF from 127.16.40.193:59259 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:35.088490 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:35.082457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:40263)
0812 18:54:35.082770 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:35.082784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:35.082885 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:35.083130 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:35.083132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:35.083320 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:35.083426 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.083432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.084247 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.084250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.085050 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.085063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.085622 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.085625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.085703 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.086266 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:35.086278 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:35.088335 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260812 18:54:35.088786 27422 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:35.082516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:40263 (local address 127.16.40.254:36241)
0812 18:54:35.082704 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:35.082707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:35.082857 (+ 150us) server_negotiation.cc:407] Connection header received
0812 18:54:35.082965 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:35.082967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:35.083013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:35.083113 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:35.083533 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.084100 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.085206 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.085446 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.085553 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.088638 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:40263: BlockingRecv error: recv got EOF from 127.16.40.194:40263 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.195@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:35.320894 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:35.315084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.195:35705)
0812 18:54:35.315360 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:35.315374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:35.315487 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:35.315772 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:35.315774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:35.316001 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:35.316110 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.316116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.316931 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.316935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.317523 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.317533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.317981 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.317984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.318052 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.318617 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:35.318633 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:35.320698 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
W20260812 18:54:35.320960 27422 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:35.315147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:35705 (local address 127.16.40.254:36241)
0812 18:54:35.315291 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:35.315294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:35.315443 (+ 149us) server_negotiation.cc:407] Connection header received
0812 18:54:35.315571 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:35.315573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:35.315623 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:35.315694 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:35.316231 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.316820 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.317634 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.317850 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.317927 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.320839 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.195:35705: BlockingRecv error: recv got EOF from 127.16.40.195:35705 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:35 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.193@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:35.896339 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:35.890749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.193:40855)
0812 18:54:35.891049 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:35.891073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:35.891164 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:35.891390 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:35.891392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:35.891580 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:35.891676 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.891681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.892452 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.892455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.893101 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:35.893112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.893718 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.893721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:35.893796 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.894322 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:35.894335 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:35.896216 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
W20260812 18:54:35.896595 27426 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:35.890878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:40855 (local address 127.16.40.254:36241)
0812 18:54:35.891159 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:35.891162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:35.891176 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:54:35.891233 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:35.891235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:35.891278 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:35.891355 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:35.891783 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.892328 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.893317 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:35.893599 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:35.893685 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:35.896356 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.193:40855: BlockingRecv error: recv got EOF from 127.16.40.193:40855 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":104,"threads_started":1}
Aug 12 18:54:36 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 18:54:36 dist-test-slave-rnrw krb5kdc[26198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.40.194@KRBTEST.COM for oryx/127.16.40.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 18:54:36.097146 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.089750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:36241 (local address 127.16.40.194:38873)
0812 18:54:36.090151 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:36.090165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:36.090272 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:36.090745 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:36.090747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:36.090950 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:36.091085 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.091092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.092225 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.092228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.093039 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.093059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.093943 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.093946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.094030 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.094706 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:36.094723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:36.096975 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.40.254:36241: Server oryx/127.16.40.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
W20260812 18:54:36.097369 27426 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:36.089897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:38873 (local address 127.16.40.254:36241)
0812 18:54:36.090099 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:36.090103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:36.090247 (+ 144us) server_negotiation.cc:407] Connection header received
0812 18:54:36.090442 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:36.090446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:36.090545 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:36.090641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:36.091246 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.092061 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.093386 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.093814 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.093941 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.097183 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.40.194:38873: BlockingRecv error: recv got EOF from 127.16.40.194:38873 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:54:36.192641 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26876
I20260812 18:54:36.202342 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27052
I20260812 18:54:36.218142 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27202
I20260812 18:54:36.228291 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 26799
2026-08-12T18:54:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18349 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T18:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:36Z Disabled control of system clock
I20260812 18:54:36.267437 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45237
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 18:54:36.386499 27431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:36.386821 27431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:36.386858 27431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:36.391234 27431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:36.391325 27431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:36.391341 27431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:36.391359 27431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:36.391376 27431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:36.396502 27431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:45237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:36.397825 27431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:36.399329 27431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:36.409471 27431 server_base.cc:1034] running on GCE node
W20260812 18:54:36.409386 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.409395 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.409317 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:36.410422 27431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:36.411782 27431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:36.413026 27431 hybrid_clock.cc:648] HybridClock initialized: now 1786560876413003 us; error 40 us; skew 500 ppm
I20260812 18:54:36.415470 27431 webserver.cc:460] Webserver started at http://127.16.40.254:33817/ using document root <none> and password file <none>
I20260812 18:54:36.416182 27431 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:36.416293 27431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:36.416601 27431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:36.418920 27431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "566e730a28364ed2aa9462d7750f776c"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "06c5df6f35b84775dae5e59c81f70dec"
server_key_iv: "efd5ed1e01bf39f74d13566188f7c8b5"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.419596 27431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "566e730a28364ed2aa9462d7750f776c"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "06c5df6f35b84775dae5e59c81f70dec"
server_key_iv: "efd5ed1e01bf39f74d13566188f7c8b5"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.423775 27431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:54:36.426723 27446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.428251 27431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:36.428469 27431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "566e730a28364ed2aa9462d7750f776c"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "06c5df6f35b84775dae5e59c81f70dec"
server_key_iv: "efd5ed1e01bf39f74d13566188f7c8b5"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.428699 27431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:36.441146 27431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:36.441999 27431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:36.442229 27431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:36.450991 27508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:45237 every 8 connection(s)
I20260812 18:54:36.450992 27431 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:45237
I20260812 18:54:36.452471 27431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:36.454890 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27431
I20260812 18:54:36.455044 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:36.455432 16547 external_mini_cluster.cc:1428] Setting key 2ceff5451f926d5ff0cfcfb6abdd27c6
I20260812 18:54:36.458065 27510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:36.465728 27510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c: Bootstrap starting.
I20260812 18:54:36.467360 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.457167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.16.40.254:45237)
0812 18:54:36.458171 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:36.458191 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:36.458295 (+ 104us) server_negotiation.cc:407] Connection header received
0812 18:54:36.459151 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:36.459179 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:36.459693 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:36.460108 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:36.460627 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.461842 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.463279 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.463837 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.463987 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.465183 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:36.465189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:36.465206 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:36.465693 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:36.465764 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:36.465795 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:36.465955 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:36.466146 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 18:54:36.466508 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":166,"threads_started":1}
I20260812 18:54:36.468443 27510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:36.469355 27510 log.cc:824] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:36.471735 27510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c: No bootstrap required, opened a new log
I20260812 18:54:36.475944 27510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } }
I20260812 18:54:36.476253 27510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:36.476341 27510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566e730a28364ed2aa9462d7750f776c, State: Initialized, Role: FOLLOWER
I20260812 18:54:36.476909 27510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } }
I20260812 18:54:36.477088 27510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:36.477182 27510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:36.477313 27510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:36.478407 27510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } }
I20260812 18:54:36.478869 27510 leader_election.cc:302] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 566e730a28364ed2aa9462d7750f776c; no voters:
I20260812 18:54:36.479246 27510 leader_election.cc:288] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:36.479512 27516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:36.479802 27516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [term 1 LEADER]: Becoming Leader. State: Replica: 566e730a28364ed2aa9462d7750f776c, State: Running, Role: LEADER
I20260812 18:54:36.480163 27516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } }
I20260812 18:54:36.480801 27510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:36.482160 27518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 566e730a28364ed2aa9462d7750f776c. Latest consensus state: current_term: 1 leader_uuid: "566e730a28364ed2aa9462d7750f776c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } } }
I20260812 18:54:36.482323 27518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:36.482679 27517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "566e730a28364ed2aa9462d7750f776c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566e730a28364ed2aa9462d7750f776c" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45237 } } }
I20260812 18:54:36.482821 27517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:36.482895 27530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:36.486822 27530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:36.494293 27530 catalog_manager.cc:1347] Generated new cluster ID: 8dc8c06b7942494cae38899b4cab5790
I20260812 18:54:36.494398 27530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:36.511662 27530 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:36.513092 27530 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:36.529793 27530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 566e730a28364ed2aa9462d7750f776c: Generated new TSK 0
I20260812 18:54:36.530814 27530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:54:36.540653 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--builtin_ntp_servers=127.16.40.212:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 18:54:36.668701 27541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:36.669080 27541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:36.669163 27541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:36.673583 27541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:36.673695 27541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:36.673797 27541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:36.679148 27541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:36.680478 27541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:36.682117 27541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:36.693111 27547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.693164 27546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.693199 27549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:36.693148 27541 server_base.cc:1034] running on GCE node
I20260812 18:54:36.693850 27541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:36.694677 27541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:36.695938 27541 hybrid_clock.cc:648] HybridClock initialized: now 1786560876695870 us; error 62 us; skew 500 ppm
I20260812 18:54:36.698603 27541 webserver.cc:460] Webserver started at http://127.16.40.193:40357/ using document root <none> and password file <none>
I20260812 18:54:36.699326 27541 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:36.699434 27541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:36.699754 27541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:36.702097 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "d6f9d45a40ab4c7484660f013c138ed2"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "13a53c7b4838cb06fe53124567a3f94a"
server_key_iv: "fcd9e1cecd91f86431b4be2037fb4bc2"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.702719 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f9d45a40ab4c7484660f013c138ed2"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "13a53c7b4838cb06fe53124567a3f94a"
server_key_iv: "fcd9e1cecd91f86431b4be2037fb4bc2"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.706974 27541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 18:54:36.710114 27556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.711576 27541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:54:36.711843 27541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "d6f9d45a40ab4c7484660f013c138ed2"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "13a53c7b4838cb06fe53124567a3f94a"
server_key_iv: "fcd9e1cecd91f86431b4be2037fb4bc2"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.711998 27541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:36.733532 27541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:36.734366 27541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:36.734596 27541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:36.735268 27541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:36.736389 27541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:36.736469 27541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.736569 27541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:36.736601 27541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.747138 27541 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:33207
I20260812 18:54:36.747232 27690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:33207 every 8 connection(s)
I20260812 18:54:36.748322 27541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:36.750157 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27541
I20260812 18:54:36.750288 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:36.750700 16547 external_mini_cluster.cc:1428] Setting key 398f16516212e12cd479386f4d89d360
I20260812 18:54:36.753319 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--builtin_ntp_servers=127.16.40.212:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 18:54:36.761245 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.750712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:38809 (local address 127.16.40.254:45237)
0812 18:54:36.750867 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:36.750872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:36.752320 (+ 1448us) server_negotiation.cc:407] Connection header received
0812 18:54:36.754642 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:36.754646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:36.754744 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:36.754832 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:36.756273 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.757183 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.758500 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.758915 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.759063 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.760371 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:36.760377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:36.760382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:36.760406 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:36.760439 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:36.760455 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:36.760596 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:36.760865 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 18:54:36.760993 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:54:36.762311 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.751280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:45237 (local address 127.16.40.193:38809)
0812 18:54:36.752145 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:36.752209 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:36.754259 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:36.755096 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:36.755110 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:36.755255 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:36.756019 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.756039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.757347 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.757350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.758347 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.758359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.759126 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.759130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.759300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.760037 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:36.760039 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:36.760090 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:36.760171 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:36.760659 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:36.760670 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:36.760677 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:54:36.760824 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 18:54:36.761151 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":174,"threads_started":1}
I20260812 18:54:36.764035 27691 heartbeater.cc:347] Connected to a master server at 127.16.40.254:45237
I20260812 18:54:36.764387 27691 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:36.765303 27691 heartbeater.cc:511] Master 127.16.40.254:45237 requested a full tablet report, sending...
I20260812 18:54:36.767341 27465 ts_manager.cc:194] Registered new tserver with Master: d6f9d45a40ab4c7484660f013c138ed2 (127.16.40.193:33207)
I20260812 18:54:36.768759 27465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.193:38809
W20260812 18:54:36.886562 27694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:36.886916 27694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:36.886991 27694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:36.891607 27694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:36.891710 27694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:36.891801 27694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:36.897068 27694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:36.898442 27694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:36.900123 27694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:36.912385 27694 server_base.cc:1034] running on GCE node
W20260812 18:54:36.912338 27699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.912345 27700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:36.912338 27702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:36.913211 27694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:36.913949 27694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:36.915160 27694 hybrid_clock.cc:648] HybridClock initialized: now 1786560876915136 us; error 38 us; skew 500 ppm
I20260812 18:54:36.917865 27694 webserver.cc:460] Webserver started at http://127.16.40.194:44527/ using document root <none> and password file <none>
I20260812 18:54:36.918496 27694 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:36.918574 27694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:36.918761 27694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:36.921195 27694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "fb9aee8a211143088c830d226e910225"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "6aecfa7223af4b2dc5e862f2f389412a"
server_key_iv: "04409cc908fb2776ac2b5807ff051dc1"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.921805 27694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "fb9aee8a211143088c830d226e910225"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "6aecfa7223af4b2dc5e862f2f389412a"
server_key_iv: "04409cc908fb2776ac2b5807ff051dc1"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.926542 27694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:36.929754 27710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.931288 27694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:54:36.931416 27694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "fb9aee8a211143088c830d226e910225"
format_stamp: "Formatted at 2026-08-12 18:54:36 on dist-test-slave-rnrw"
server_key: "6aecfa7223af4b2dc5e862f2f389412a"
server_key_iv: "04409cc908fb2776ac2b5807ff051dc1"
server_key_version: "encryptionkey@0"
I20260812 18:54:36.931531 27694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:36.953842 27694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:36.954635 27694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:36.954818 27694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:36.955534 27694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:36.956746 27694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:36.956802 27694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.956851 27694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:36.956866 27694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:36.967176 27694 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:45105
I20260812 18:54:36.967216 27841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:45105 every 8 connection(s)
I20260812 18:54:36.968236 27694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:36.973198 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27694
I20260812 18:54:36.973352 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:36.973760 16547 external_mini_cluster.cc:1428] Setting key 40c6d05809856107efc248d8d9a36b00
I20260812 18:54:36.976079 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--builtin_ntp_servers=127.16.40.212:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 18:54:36.979627 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.970400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:36089 (local address 127.16.40.254:45237)
0812 18:54:36.970585 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:36.970590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:36.971874 (+ 1284us) server_negotiation.cc:407] Connection header received
0812 18:54:36.973528 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:36.973532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:36.973604 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:36.973708 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:36.974911 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.975632 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.977145 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.977537 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.977666 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.979123 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:36.979126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:36.979128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:36.979142 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:36.979162 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:36.979171 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:36.979231 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:36.979430 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 18:54:36.979492 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 18:54:36.980967 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:36.970883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:45237 (local address 127.16.40.194:36089)
0812 18:54:36.971689 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:36.971735 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:36.973224 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:36.973917 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:36.973924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:36.974016 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:36.974752 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.974774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.975785 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.975789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.977016 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:36.977027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:36.977702 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:36.977705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:36.977973 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:36.978756 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:36.978758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:36.978822 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:36.978924 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:36.979262 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:36.979274 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:36.979282 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 18:54:36.979430 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 18:54:36.979775 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
I20260812 18:54:36.982465 27842 heartbeater.cc:347] Connected to a master server at 127.16.40.254:45237
I20260812 18:54:36.982829 27842 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:36.983788 27842 heartbeater.cc:511] Master 127.16.40.254:45237 requested a full tablet report, sending...
I20260812 18:54:36.985174 27465 ts_manager.cc:194] Registered new tserver with Master: fb9aee8a211143088c830d226e910225 (127.16.40.194:45105)
I20260812 18:54:36.985809 27465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.194:36089
W20260812 18:54:37.112339 27845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:37.112772 27845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:37.112820 27845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:37.117403 27845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:37.117480 27845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:37.117540 27845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:37.122944 27845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:37.124248 27845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:37.125882 27845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:37.136817 27850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.136859 27851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.136864 27854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:37.137606 27845 server_base.cc:1034] running on GCE node
I20260812 18:54:37.138121 27845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:37.138880 27845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:37.140128 27845 hybrid_clock.cc:648] HybridClock initialized: now 1786560877140089 us; error 32 us; skew 500 ppm
I20260812 18:54:37.142386 27845 webserver.cc:460] Webserver started at http://127.16.40.195:37613/ using document root <none> and password file <none>
I20260812 18:54:37.143155 27845 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:37.143263 27845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:37.143534 27845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:37.146065 27845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "071eff6d9269405bbbeb62b8266e63c4"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "bab99ec70f0874eaed6120f400dc4cdb"
server_key_iv: "9bd4ff073867af4a6f221393a625c75d"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.146746 27845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "071eff6d9269405bbbeb62b8266e63c4"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "bab99ec70f0874eaed6120f400dc4cdb"
server_key_iv: "9bd4ff073867af4a6f221393a625c75d"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.151372 27845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:54:37.154464 27861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.156066 27845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:37.156237 27845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "071eff6d9269405bbbeb62b8266e63c4"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "bab99ec70f0874eaed6120f400dc4cdb"
server_key_iv: "9bd4ff073867af4a6f221393a625c75d"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.156386 27845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:37.174103 27845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:37.174917 27845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:37.175156 27845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:37.175832 27845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:37.176924 27845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:37.177000 27845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.177078 27845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:37.177122 27845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.187318 27845 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:38949
I20260812 18:54:37.187367 27989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:38949 every 8 connection(s)
I20260812 18:54:37.188421 27845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:37.196022 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 27845
I20260812 18:54:37.196164 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:37.196524 16547 external_mini_cluster.cc:1428] Setting key 9093b4ed25225ec0c74b0ade2af666f1
I20260812 18:54:37.200353 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.190870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:54805 (local address 127.16.40.254:45237)
0812 18:54:37.191065 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:37.191071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:37.192359 (+ 1288us) server_negotiation.cc:407] Connection header received
0812 18:54:37.194029 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:37.194033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:37.194090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:37.194188 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:37.195440 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:37.196308 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:37.198022 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:37.198333 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:37.198443 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:37.199888 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:37.199891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:37.199893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:37.199913 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:37.199935 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:37.199943 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:37.200005 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:37.200164 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 18:54:37.200224 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 18:54:37.201468 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.191318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:45237 (local address 127.16.40.195:54805)
0812 18:54:37.192144 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:37.192200 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:37.193754 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:37.194438 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:37.194446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:37.194602 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:37.195253 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:37.195274 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:37.196480 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:37.196483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:37.197884 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:37.197895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:37.198534 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:37.198539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:37.198766 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:37.199599 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:37.199600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:37.199647 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:37.199725 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:37.200024 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:37.200032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:37.200037 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 18:54:37.200145 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 18:54:37.200405 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":139,"threads_started":1}
I20260812 18:54:37.202948 27990 heartbeater.cc:347] Connected to a master server at 127.16.40.254:45237
I20260812 18:54:37.203282 27990 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:37.203994 27990 heartbeater.cc:511] Master 127.16.40.254:45237 requested a full tablet report, sending...
I20260812 18:54:37.205387 27465 ts_manager.cc:194] Registered new tserver with Master: 071eff6d9269405bbbeb62b8266e63c4 (127.16.40.195:38949)
I20260812 18:54:37.206056 27465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.195:54805
I20260812 18:54:37.211741 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 18:54:37.215139 27513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:37.214049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.16.40.254:45237)
0812 18:54:37.214247 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:37.214251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:37.214341 (+ 90us) server_negotiation.cc:407] Connection header received
0812 18:54:37.214498 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:37.214501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:37.214548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:37.214613 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:37.214943 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41134: BlockingRecv error: recv got EOF from 127.0.0.1:41134 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 18:54:37.215250 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27541
I20260812 18:54:37.224431 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27694
I20260812 18:54:37.234123 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27845
I20260812 18:54:37.245960 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 27431
2026-08-12T18:54:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1016 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T18:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:37Z Disabled control of system clock
I20260812 18:54:37.283774 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35379
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:35379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 18:54:37.412683 28004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:37.412998 28004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:37.413038 28004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:37.417165 28004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:37.417236 28004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:37.417253 28004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:37.417270 28004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:37.417286 28004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:37.422386 28004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:35379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:37.423615 28004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:37.425151 28004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:37.434731 28013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.434718 28011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.434718 28010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:37.435096 28004 server_base.cc:1034] running on GCE node
I20260812 18:54:37.435798 28004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:37.437137 28004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:37.438347 28004 hybrid_clock.cc:648] HybridClock initialized: now 1786560877438334 us; error 33 us; skew 500 ppm
I20260812 18:54:37.440744 28004 webserver.cc:460] Webserver started at http://127.16.40.254:44677/ using document root <none> and password file <none>
I20260812 18:54:37.441421 28004 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:37.441527 28004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:37.441787 28004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:37.444079 28004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "220f277a018d47dfbe7552be36370c54"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "52616d3210469305fb3e592c9c891dfa"
server_key_iv: "2f57491120255448a3a5162fa76f10ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.444814 28004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "220f277a018d47dfbe7552be36370c54"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "52616d3210469305fb3e592c9c891dfa"
server_key_iv: "2f57491120255448a3a5162fa76f10ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.449290 28004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:37.452907 28022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.454713 28004 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:54:37.454944 28004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "220f277a018d47dfbe7552be36370c54"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "52616d3210469305fb3e592c9c891dfa"
server_key_iv: "2f57491120255448a3a5162fa76f10ab"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.455123 28004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:37.460696 28004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:37.461565 28004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:37.461798 28004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:37.469888 28004 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:35379
I20260812 18:54:37.469902 28082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:35379 every 8 connection(s)
I20260812 18:54:37.471041 28004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:37.471454 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28004
I20260812 18:54:37.471608 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:37.471971 16547 external_mini_cluster.cc:1428] Setting key 784b47183a6cb92fd1147306b6a337d0
I20260812 18:54:37.475123 28083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:37.479817 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.473638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.16.40.254:35379)
0812 18:54:37.474876 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:37.474901 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:37.474955 (+ 54us) server_negotiation.cc:407] Connection header received
0812 18:54:37.475955 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:37.475982 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:37.476507 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:37.477008 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:37.477934 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:37.477941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:37.477959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:37.477972 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:37.478049 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:37.478076 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:37.478274 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:37.478353 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 18:54:37.478683 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":534,"threads_started":1}
I20260812 18:54:37.483834 28083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54: Bootstrap starting.
I20260812 18:54:37.486698 28083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:37.487773 28083 log.cc:824] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:37.490671 28083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54: No bootstrap required, opened a new log
I20260812 18:54:37.494736 28083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } }
I20260812 18:54:37.495118 28083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:37.495190 28083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220f277a018d47dfbe7552be36370c54, State: Initialized, Role: FOLLOWER
I20260812 18:54:37.495702 28083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } }
I20260812 18:54:37.495880 28083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:37.495957 28083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:37.496065 28083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:37.497272 28083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } }
I20260812 18:54:37.497689 28083 leader_election.cc:302] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220f277a018d47dfbe7552be36370c54; no voters:
I20260812 18:54:37.498081 28083 leader_election.cc:288] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:37.498299 28088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:37.498557 28088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [term 1 LEADER]: Becoming Leader. State: Replica: 220f277a018d47dfbe7552be36370c54, State: Running, Role: LEADER
I20260812 18:54:37.498984 28088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } }
I20260812 18:54:37.499608 28083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:37.500936 28091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 220f277a018d47dfbe7552be36370c54. Latest consensus state: current_term: 1 leader_uuid: "220f277a018d47dfbe7552be36370c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } } }
I20260812 18:54:37.501108 28091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:37.501657 28089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "220f277a018d47dfbe7552be36370c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220f277a018d47dfbe7552be36370c54" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35379 } } }
I20260812 18:54:37.501806 28098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:37.502900 28089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:37.505762 28098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:37.512926 28098 catalog_manager.cc:1347] Generated new cluster ID: ea35e4fcaa064d238d0e245ff7d3dd3a
I20260812 18:54:37.513048 28098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:37.523175 28098 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:37.523591 28098 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:37.533329 28098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 220f277a018d47dfbe7552be36370c54: Generated new TSK 0
I20260812 18:54:37.534190 28098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:54:37.539461 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--builtin_ntp_servers=127.16.40.212:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 18:54:37.664523 28110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:37.664844 28110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:37.664886 28110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:37.669098 28110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:37.669165 28110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:37.669255 28110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:37.674230 28110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:37.675458 28110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:37.676960 28110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:37.686817 28116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.686827 28119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.686842 28115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:37.687323 28110 server_base.cc:1034] running on GCE node
I20260812 18:54:37.687826 28110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:37.688535 28110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:37.689791 28110 hybrid_clock.cc:648] HybridClock initialized: now 1786560877689770 us; error 33 us; skew 500 ppm
I20260812 18:54:37.692056 28110 webserver.cc:460] Webserver started at http://127.16.40.193:39267/ using document root <none> and password file <none>
I20260812 18:54:37.692812 28110 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:37.692898 28110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:37.693172 28110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:37.695513 28110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "bf39d2c2eef649bcb27574c312ef5ec7"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "3260b01732c38ac557ca5945381d5c91"
server_key_iv: "a2ffa9ae7fc60d08b4b4ffa851c07408"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.696138 28110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "bf39d2c2eef649bcb27574c312ef5ec7"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "3260b01732c38ac557ca5945381d5c91"
server_key_iv: "a2ffa9ae7fc60d08b4b4ffa851c07408"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.700692 28110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:54:37.703812 28127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.705322 28110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:37.705506 28110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "bf39d2c2eef649bcb27574c312ef5ec7"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "3260b01732c38ac557ca5945381d5c91"
server_key_iv: "a2ffa9ae7fc60d08b4b4ffa851c07408"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.705655 28110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:37.710884 28110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:37.711603 28110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:37.711810 28110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:37.712440 28110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:37.713572 28110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:37.713657 28110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.713737 28110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:37.713780 28110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.723414 28110 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:46685
I20260812 18:54:37.723441 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:46685 every 8 connection(s)
I20260812 18:54:37.724452 28110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:37.727933 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28110
I20260812 18:54:37.728094 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:37.728447 16547 external_mini_cluster.cc:1428] Setting key 184a9a3d18e9a0ef7de0736f123776bb
I20260812 18:54:37.730919 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--builtin_ntp_servers=127.16.40.212:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 18:54:37.731977 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.726718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:53593 (local address 127.16.40.254:35379)
0812 18:54:37.726878 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:37.726883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:37.728041 (+ 1158us) server_negotiation.cc:407] Connection header received
0812 18:54:37.729761 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:37.729765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:37.729808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:37.729895 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:37.731435 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:37.731445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:37.731448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:37.731462 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:37.731486 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:37.731494 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:37.731597 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:37.731773 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 18:54:37.731825 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 18:54:37.733137 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.727094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35379 (local address 127.16.40.193:53593)
0812 18:54:37.727864 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:37.727908 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:37.729526 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:37.730193 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:37.730200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:37.730296 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:37.731036 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:37.731041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:37.731113 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:37.731230 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:37.731596 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:37.731609 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:37.731619 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 18:54:37.731824 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 18:54:37.732121 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":115,"threads_started":1}
I20260812 18:54:37.734658 28261 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35379
I20260812 18:54:37.734951 28261 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:37.735487 28261 heartbeater.cc:511] Master 127.16.40.254:35379 requested a full tablet report, sending...
I20260812 18:54:37.737586 28041 ts_manager.cc:194] Registered new tserver with Master: bf39d2c2eef649bcb27574c312ef5ec7 (127.16.40.193:46685)
W20260812 18:54:37.852898 28264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:37.853225 28264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:37.853268 28264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:37.857774 28264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:37.857861 28264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:37.858062 28264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:37.864020 28264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:37.865355 28264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:37.866864 28264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:37.877463 28271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.877451 28270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:37.877451 28274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:37.878254 28264 server_base.cc:1034] running on GCE node
I20260812 18:54:37.878801 28264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:37.879602 28264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:37.880777 28264 hybrid_clock.cc:648] HybridClock initialized: now 1786560877880717 us; error 77 us; skew 500 ppm
I20260812 18:54:37.883106 28264 webserver.cc:460] Webserver started at http://127.16.40.194:41465/ using document root <none> and password file <none>
I20260812 18:54:37.883783 28264 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:37.883853 28264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:37.884043 28264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:37.886374 28264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "265c62a5f3da4b2bbe9f0e55f47f7285"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "4cb40ad54d247f98af22f4bf12302e8f"
server_key_iv: "a130ad8b022df4c692d595a77a340683"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.886942 28264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "265c62a5f3da4b2bbe9f0e55f47f7285"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "4cb40ad54d247f98af22f4bf12302e8f"
server_key_iv: "a130ad8b022df4c692d595a77a340683"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.891424 28264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:54:37.894747 28280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.896353 28264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:37.896502 28264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "265c62a5f3da4b2bbe9f0e55f47f7285"
format_stamp: "Formatted at 2026-08-12 18:54:37 on dist-test-slave-rnrw"
server_key: "4cb40ad54d247f98af22f4bf12302e8f"
server_key_iv: "a130ad8b022df4c692d595a77a340683"
server_key_version: "encryptionkey@0"
I20260812 18:54:37.896711 28264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:37.902381 28264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:37.903208 28264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:37.903391 28264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:37.903988 28264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:37.905115 28264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:37.905169 28264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.905212 28264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:37.905228 28264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:37.915385 28264 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:34685
I20260812 18:54:37.915409 28411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:34685 every 8 connection(s)
I20260812 18:54:37.916430 28264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:37.919539 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28264
I20260812 18:54:37.919682 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:37.920010 16547 external_mini_cluster.cc:1428] Setting key 669e20ff670e55b28508de95381a04a5
I20260812 18:54:37.922358 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--builtin_ntp_servers=127.16.40.212:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 18:54:37.923605 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.918466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:44439 (local address 127.16.40.254:35379)
0812 18:54:37.918670 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:37.918674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:37.919885 (+ 1211us) server_negotiation.cc:407] Connection header received
0812 18:54:37.921664 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:37.921666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:37.921731 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:37.921873 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:37.923026 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:37.923032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:37.923039 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:37.923090 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:37.923118 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:37.923132 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:37.923235 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:37.923365 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 18:54:37.923417 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 18:54:37.924700 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:37.918912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35379 (local address 127.16.40.194:44439)
0812 18:54:37.919702 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:37.919762 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:37.921437 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:37.922030 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:37.922038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:37.922127 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:37.922754 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:37.922756 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:37.922799 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:37.922869 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:37.923225 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:37.923232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:37.923237 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 18:54:37.923376 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 18:54:37.923583 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":136,"threads_started":1}
I20260812 18:54:37.925866 28412 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35379
I20260812 18:54:37.926126 28412 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:37.926643 28412 heartbeater.cc:511] Master 127.16.40.254:35379 requested a full tablet report, sending...
I20260812 18:54:37.927860 28041 ts_manager.cc:194] Registered new tserver with Master: 265c62a5f3da4b2bbe9f0e55f47f7285 (127.16.40.194:34685)
W20260812 18:54:38.046530 28417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:38.046887 28417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:38.046960 28417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:38.051383 28417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:38.051498 28417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:38.051599 28417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:38.056999 28417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:38.058307 28417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:38.059950 28417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:38.070704 28425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:38.070698 28422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:38.070824 28417 server_base.cc:1034] running on GCE node
W20260812 18:54:38.070724 28423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:38.071516 28417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:38.072314 28417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:38.073511 28417 hybrid_clock.cc:648] HybridClock initialized: now 1786560878073474 us; error 59 us; skew 500 ppm
I20260812 18:54:38.075610 28417 webserver.cc:460] Webserver started at http://127.16.40.195:45151/ using document root <none> and password file <none>
I20260812 18:54:38.076220 28417 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:38.076289 28417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:38.076472 28417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:38.078791 28417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "03ec7bf12feb4874afc51aefd38117ee"
format_stamp: "Formatted at 2026-08-12 18:54:38 on dist-test-slave-rnrw"
server_key: "e2cf35efc8e0a3f5f5f3cc36419c3c2b"
server_key_iv: "351204652f66b38e8416ba158a8fdea1"
server_key_version: "encryptionkey@0"
I20260812 18:54:38.079340 28417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "03ec7bf12feb4874afc51aefd38117ee"
format_stamp: "Formatted at 2026-08-12 18:54:38 on dist-test-slave-rnrw"
server_key: "e2cf35efc8e0a3f5f5f3cc36419c3c2b"
server_key_iv: "351204652f66b38e8416ba158a8fdea1"
server_key_version: "encryptionkey@0"
I20260812 18:54:38.083463 28417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:54:38.086668 28433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:38.088168 28417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:54:38.088418 28417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "03ec7bf12feb4874afc51aefd38117ee"
format_stamp: "Formatted at 2026-08-12 18:54:38 on dist-test-slave-rnrw"
server_key: "e2cf35efc8e0a3f5f5f3cc36419c3c2b"
server_key_iv: "351204652f66b38e8416ba158a8fdea1"
server_key_version: "encryptionkey@0"
I20260812 18:54:38.088613 28417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:38.094043 28417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:38.095322 28417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:38.095551 28417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:38.096237 28417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:38.097414 28417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:38.097467 28417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:38.097510 28417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:38.097569 28417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:38.107275 28417 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:43717
I20260812 18:54:38.107323 28561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:43717 every 8 connection(s)
I20260812 18:54:38.108340 28417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:38.110978 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28417
I20260812 18:54:38.111099 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:38.111405 16547 external_mini_cluster.cc:1428] Setting key c8e51fc5e2ca89dfdfd9e61c6bb61601
I20260812 18:54:38.116262 28086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:38.110746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:34281 (local address 127.16.40.254:35379)
0812 18:54:38.110929 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:38.110934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:38.112792 (+ 1858us) server_negotiation.cc:407] Connection header received
0812 18:54:38.114377 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:38.114381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:38.114430 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:38.114497 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:38.115688 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:38.115692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:38.115695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:54:38.115716 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:38.115740 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:54:38.115749 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:38.115851 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:38.116004 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 18:54:38.116057 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 18:54:38.117333 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:38.111290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35379 (local address 127.16.40.195:34281)
0812 18:54:38.112505 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:38.112613 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:38.114130 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:38.114686 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:38.114693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:38.114797 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:38.115401 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:54:38.115403 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:38.115450 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:54:38.115530 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:38.115834 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:38.115844 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:38.115850 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 18:54:38.116099 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 18:54:38.116430 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":197,"threads_started":1}
I20260812 18:54:38.118816 28562 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35379
I20260812 18:54:38.119091 28562 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:38.119601 28562 heartbeater.cc:511] Master 127.16.40.254:35379 requested a full tablet report, sending...
I20260812 18:54:38.120998 28041 ts_manager.cc:194] Registered new tserver with Master: 03ec7bf12feb4874afc51aefd38117ee (127.16.40.195:43717)
I20260812 18:54:38.127059 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 18:54:38.130470 28086 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:54:38.129303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.16.40.254:35379)
0812 18:54:38.129566 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:38.129571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:38.129705 (+ 134us) server_negotiation.cc:407] Connection header received
0812 18:54:38.129884 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:38.129891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:38.129940 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:38.130058 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:38.130331 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49346: BlockingRecv error: recv got EOF from 127.0.0.1:49346 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 18:54:38.130903 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28110
I20260812 18:54:38.145728 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28264
I20260812 18:54:38.163569 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28417
I20260812 18:54:38.176083 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28004
2026-08-12T18:54:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (929 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:38 dist-test-slave-rnrw krb5kdc[28572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:38 dist-test-slave-rnrw krb5kdc[28572](info): set up 2 sockets
Aug 12 18:54:38 dist-test-slave-rnrw krb5kdc[28572](info): commencing operation
krb5kdc: starting...
W20260812 18:54:39.337638 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:39 dist-test-slave-rnrw krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:54:39.484511 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:44079
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:44079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:39.608706 28587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:39.609086 28587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:39.609169 28587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:39.613569 28587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:54:39.613703 28587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:39.613750 28587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:39.613816 28587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:54:39.613854 28587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:54:39.619715 28587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:44079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:44079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:39.621094 28587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:39.622254 28587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:39.631592 28592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:39.631654 28593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:39.631654 28595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:39.632108 28587 server_base.cc:1034] running on GCE node
I20260812 18:54:39.632850 28587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:39.634114 28587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:39.635334 28587 hybrid_clock.cc:648] HybridClock initialized: now 1786560879635288 us; error 62 us; skew 500 ppm
Aug 12 18:54:39 dist-test-slave-rnrw krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560879, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:39.638953 28587 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:54:39.640465 28587 webserver.cc:460] Webserver started at http://127.16.40.254:40219/ using document root <none> and password file <none>
I20260812 18:54:39.641196 28587 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:39.641294 28587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:39.641577 28587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:39.643959 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "808605f63f9d4e63a4488ea75dbccfe2"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "e667625f1c5615cb514c9efac5a89443"
server_key_iv: "62624fbdda89b253ab16356e7c58c82e"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.644515 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "808605f63f9d4e63a4488ea75dbccfe2"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "e667625f1c5615cb514c9efac5a89443"
server_key_iv: "62624fbdda89b253ab16356e7c58c82e"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.649117 28587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:54:39.652441 28603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:39.654085 28587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:39.654328 28587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "808605f63f9d4e63a4488ea75dbccfe2"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "e667625f1c5615cb514c9efac5a89443"
server_key_iv: "62624fbdda89b253ab16356e7c58c82e"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.654484 28587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:39.669373 28587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:39.673458 28587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:39.673693 28587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:39.681196 28587 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:44079
I20260812 18:54:39.681609 28661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:44079 every 8 connection(s)
I20260812 18:54:39.682296 28587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:54:39.683358 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28587
I20260812 18:54:39.683578 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:54:39.683888 16547 external_mini_cluster.cc:1428] Setting key cc4d4875367c3fe17b66b4d0ef82be69
I20260812 18:54:39.686124 28662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:39.694566 28662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: Bootstrap starting.
Aug 12 18:54:39 dist-test-slave-rnrw krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:39.697788 28662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:39.698969 28662 log.cc:824] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:39.701725 28662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: No bootstrap required, opened a new log
I20260812 18:54:39.702092 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:39.685631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.16.40.254:44079)
0812 18:54:39.686126 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:39.686149 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:39.686203 (+ 54us) server_negotiation.cc:407] Connection header received
0812 18:54:39.687174 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:39.687200 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:39.687718 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:39.688159 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:39.689025 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:39.690836 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:39.691916 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:39.692769 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:39.692987 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:39.696196 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:39.696202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:39.696218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:39.696255 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:39.698507 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:39.699184 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:39.699193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:39.699201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:39.699352 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:39.699744 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:39.699747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:39.699748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:39.700362 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:39.700528 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:39.700890 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 18:54:39.701253 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":130,"threads_started":1}
I20260812 18:54:39.705994 28662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } }
I20260812 18:54:39.706238 28662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:39.706308 28662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808605f63f9d4e63a4488ea75dbccfe2, State: Initialized, Role: FOLLOWER
I20260812 18:54:39.706744 28662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } }
I20260812 18:54:39.706879 28662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:54:39.706921 28662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:54:39.707023 28662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:39.708133 28662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } }
I20260812 18:54:39.708505 28662 leader_election.cc:302] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 808605f63f9d4e63a4488ea75dbccfe2; no voters:
I20260812 18:54:39.708897 28662 leader_election.cc:288] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:54:39.709090 28669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:39.709376 28669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [term 1 LEADER]: Becoming Leader. State: Replica: 808605f63f9d4e63a4488ea75dbccfe2, State: Running, Role: LEADER
I20260812 18:54:39.709798 28669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } }
I20260812 18:54:39.710493 28662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:54:39.711763 28671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808605f63f9d4e63a4488ea75dbccfe2. Latest consensus state: current_term: 1 leader_uuid: "808605f63f9d4e63a4488ea75dbccfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } } }
I20260812 18:54:39.711947 28671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:39.711762 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "808605f63f9d4e63a4488ea75dbccfe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808605f63f9d4e63a4488ea75dbccfe2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44079 } } }
I20260812 18:54:39.712100 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2 [sys.catalog]: This master's current role is: LEADER
I20260812 18:54:39.713423 28679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:54:39.716535 28679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:54:39.723174 28679 catalog_manager.cc:1347] Generated new cluster ID: 48671187bf4b46578802734b0e916e70
I20260812 18:54:39.723289 28679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:54:39.738498 28679 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:54:39.739559 28679 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:54:39.748109 28679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: Generated new TSK 0
I20260812 18:54:39.749027 28679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:54:39.789526 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:39.914983 28693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:39.915331 28693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:39.915438 28693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:39.919773 28693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:39.919891 28693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:39.919993 28693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:54:39.928112 28693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:39.930068 28693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:39.931425 28693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:39.941715 28702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:39.941730 28700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:39.941782 28693 server_base.cc:1034] running on GCE node
W20260812 18:54:39.941745 28699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:39.942426 28693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:39.943122 28693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:39.944314 28693 hybrid_clock.cc:648] HybridClock initialized: now 1786560879944297 us; error 28 us; skew 500 ppm
Aug 12 18:54:39 dist-test-slave-rnrw krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560879, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:39.952477 28693 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:54:39.953605 28693 webserver.cc:460] Webserver started at http://127.16.40.193:40689/ using document root <none> and password file <none>
I20260812 18:54:39.954241 28693 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:39.954339 28693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:39.954610 28693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:39.956956 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "8490783ebc674dc593ca41e7be480460"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "9e8240956ba3abc1eed0b956c8f74c5a"
server_key_iv: "a9f7008cb13084dc6dde9010069fb05b"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.957551 28693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "8490783ebc674dc593ca41e7be480460"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "9e8240956ba3abc1eed0b956c8f74c5a"
server_key_iv: "a9f7008cb13084dc6dde9010069fb05b"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.961781 28693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:54:39.964313 28709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:39.965760 28693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:39.965906 28693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "8490783ebc674dc593ca41e7be480460"
format_stamp: "Formatted at 2026-08-12 18:54:39 on dist-test-slave-rnrw"
server_key: "9e8240956ba3abc1eed0b956c8f74c5a"
server_key_iv: "a9f7008cb13084dc6dde9010069fb05b"
server_key_version: "encryptionkey@0"
I20260812 18:54:39.966027 28693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:39.988639 28693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:39.991852 28693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:39.992089 28693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:39.992822 28693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:39.993750 28693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:39.993825 28693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:39.993901 28693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:39.993944 28693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.004284 28693 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43003
I20260812 18:54:40.004316 28837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43003 every 8 connection(s)
I20260812 18:54:40.005343 28693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:54:40.008867 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28693
I20260812 18:54:40.009012 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:54:40.009292 16547 external_mini_cluster.cc:1428] Setting key b4a86abf418981ebc4fa937ce2dd6670
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560879, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:54:40.024072 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.007511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:42329 (local address 127.16.40.254:44079)
0812 18:54:40.007703 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.007707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.008803 (+ 1096us) server_negotiation.cc:407] Connection header received
0812 18:54:40.010511 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.010514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.010570 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.010684 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:40.012308 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.013195 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.015048 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.015491 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.015639 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.019067 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:40.019074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:40.019077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:40.019119 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:40.021191 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.021967 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.021970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.021972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.022061 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.022472 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.022476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.022478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.022895 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:40.022997 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.023787 (+ 790us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.023930 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:54:40.025242 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.007930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44079 (local address 127.16.40.193:42329)
0812 18:54:40.008572 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.008671 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.010261 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.010937 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.010963 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.011508 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:40.012149 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.012166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.013366 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.013372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.014867 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.014879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.015700 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.015704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.015883 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.016745 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:40.016779 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:40.018816 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:40.021381 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.021389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.021407 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.021834 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.022166 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.022169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.022170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.022341 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.023020 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:40.023027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:40.023348 (+ 321us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.023532 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.023978 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":138,"threads_started":1}
I20260812 18:54:40.027215 28838 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44079
I20260812 18:54:40.027613 28838 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:40.028388 28838 heartbeater.cc:511] Master 127.16.40.254:44079 requested a full tablet report, sending...
I20260812 18:54:40.030380 28621 ts_manager.cc:194] Registered new tserver with Master: 8490783ebc674dc593ca41e7be480460 (127.16.40.193:43003)
I20260812 18:54:40.031816 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:42329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:54:40.045398 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:40.170107 28843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:40.170516 28843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:40.170607 28843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:40.175154 28843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:40.175284 28843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:40.175390 28843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:54:40.181427 28843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:40.182717 28843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:40.183893 28843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:54:40.196156 28843 server_base.cc:1034] running on GCE node
W20260812 18:54:40.196197 28849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:40.196200 28851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:40.196486 28848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:40.197154 28843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:40.197980 28843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:40.199198 28843 hybrid_clock.cc:648] HybridClock initialized: now 1786560880199158 us; error 57 us; skew 500 ppm
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:40.209559 28843 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:54:40.210822 28843 webserver.cc:460] Webserver started at http://127.16.40.194:38437/ using document root <none> and password file <none>
I20260812 18:54:40.211650 28843 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:40.211755 28843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:40.212019 28843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:40.214632 28843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "9ca84eeaf93f4eeb91ea66aa42b11724"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "ed36f49dbd3dc11f9b2fb37612a6f90b"
server_key_iv: "13b5e840b30e4aa88076e5336acd3244"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.215302 28843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "9ca84eeaf93f4eeb91ea66aa42b11724"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "ed36f49dbd3dc11f9b2fb37612a6f90b"
server_key_iv: "13b5e840b30e4aa88076e5336acd3244"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.220010 28843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 18:54:40.223240 28858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.225355 28843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:54:40.225567 28843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "9ca84eeaf93f4eeb91ea66aa42b11724"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "ed36f49dbd3dc11f9b2fb37612a6f90b"
server_key_iv: "13b5e840b30e4aa88076e5336acd3244"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.225724 28843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:40.237768 28843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:40.242014 28843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:40.242286 28843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:40.243110 28843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:40.244151 28843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:40.244239 28843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.244319 28843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:40.244367 28843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.255877 28843 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:43699
I20260812 18:54:40.255924 28982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:43699 every 8 connection(s)
I20260812 18:54:40.257519 28843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:54:40.265321 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28843
I20260812 18:54:40.265478 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:54:40.265811 16547 external_mini_cluster.cc:1428] Setting key c71cdeb79717eb35b105995c388cd321
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:40.276526 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.259878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:36227 (local address 127.16.40.254:44079)
0812 18:54:40.260088 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.260093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.261288 (+ 1195us) server_negotiation.cc:407] Connection header received
0812 18:54:40.263088 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.263093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.263170 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.263304 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:40.265011 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.265935 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.267572 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.267899 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.268045 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.271211 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:40.271217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:40.271222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:40.271263 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:40.273467 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.274267 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.274274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.274279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.274397 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.274915 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.274922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.274926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.275264 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:40.275396 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.276132 (+ 736us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.276308 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 18:54:40.277447 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.260379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44079 (local address 127.16.40.194:36227)
0812 18:54:40.261097 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.261150 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.262787 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.263553 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.263562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.264041 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:40.264750 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.264771 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.266142 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.266145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.267353 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.267367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.268174 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.268177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.268353 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.269116 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:40.269138 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:40.270977 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:40.273694 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.273707 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.273739 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.274107 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.274539 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.274542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.274543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.274777 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.275489 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:40.275496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:40.275805 (+ 309us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.276013 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.276349 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":256,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:54:40.279083 28983 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44079
I20260812 18:54:40.279425 28983 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:40.280120 28983 heartbeater.cc:511] Master 127.16.40.254:44079 requested a full tablet report, sending...
I20260812 18:54:40.281797 28621 ts_manager.cc:194] Registered new tserver with Master: 9ca84eeaf93f4eeb91ea66aa42b11724 (127.16.40.194:43699)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:54:40.282598 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:36227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:54:40.304250 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:54:40.443095 28988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:54:40.443419 28988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:54:40.443467 28988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:54:40.447896 28988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:54:40.447999 28988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:54:40.448076 28988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:54:40.454845 28988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:54:40.456212 28988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:54:40.457652 28988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:54:40.468786 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:40.468791 28993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:54:40.468786 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:54:40.469326 28988 server_base.cc:1034] running on GCE node
I20260812 18:54:40.469856 28988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:54:40.470741 28988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:54:40.471939 28988 hybrid_clock.cc:648] HybridClock initialized: now 1786560880471896 us; error 57 us; skew 500 ppm
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[28572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:54:40.475277 28988 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:54:40.476259 28988 webserver.cc:460] Webserver started at http://127.16.40.195:38581/ using document root <none> and password file <none>
I20260812 18:54:40.477082 28988 fs_manager.cc:359] Metadata directory not provided
I20260812 18:54:40.477175 28988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:54:40.477397 28988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:54:40.479677 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "d2885a72fc094ae39321a477b2cd2336"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "3d040714a0593e6d898cc7e72d829129"
server_key_iv: "1e24ecd1337050e2ab01e47eedc63520"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.480259 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "d2885a72fc094ae39321a477b2cd2336"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "3d040714a0593e6d898cc7e72d829129"
server_key_iv: "1e24ecd1337050e2ab01e47eedc63520"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.484405 28988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:54:40.487246 29003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.488826 28988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:54:40.488950 28988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "d2885a72fc094ae39321a477b2cd2336"
format_stamp: "Formatted at 2026-08-12 18:54:40 on dist-test-slave-rnrw"
server_key: "3d040714a0593e6d898cc7e72d829129"
server_key_iv: "1e24ecd1337050e2ab01e47eedc63520"
server_key_version: "encryptionkey@0"
I20260812 18:54:40.489059 28988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:54:40.513563 28988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:54:40.517695 28988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:54:40.518008 28988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:54:40.518785 28988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:54:40.519724 28988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:54:40.519774 28988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.519820 28988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:54:40.519836 28988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:54:40.531354 28988 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:34057
I20260812 18:54:40.531443 29129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:34057 every 8 connection(s)
I20260812 18:54:40.532495 28988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:54:40.533752 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 28988
I20260812 18:54:40.533910 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:54:40.534240 16547 external_mini_cluster.cc:1428] Setting key 172e2d3e8a731447a3a6edcd07a8bb03
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[28572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560880, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:54:40.552034 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.535091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:40221 (local address 127.16.40.254:44079)
0812 18:54:40.535337 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.535342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.536595 (+ 1253us) server_negotiation.cc:407] Connection header received
0812 18:54:40.538447 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.538451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.538526 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.538612 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:40.540082 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.541081 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.542860 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.543184 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.543548 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.547073 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:40.547077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:40.547080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:40.547114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:40.549228 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.550064 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.550068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.550070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.550172 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.550693 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.550696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.550697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.550927 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:40.551034 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.551725 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.551864 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260812 18:54:40.553079 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.535674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44079 (local address 127.16.40.195:40221)
0812 18:54:40.536361 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.536424 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.538193 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.538841 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.538850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.539306 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:54:40.539891 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.539907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.541278 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.541282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.542542 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.542558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.543554 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.543559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.543802 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.544891 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:54:40.544913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:54:40.546799 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:54:40.549412 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.549421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.549447 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.549872 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.550301 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:54:40.550311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:54:40.550313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:54:40.550544 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:54:40.551133 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:54:40.551139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:54:40.551444 (+ 305us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.551641 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.551964 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":154,"threads_started":1}
I20260812 18:54:40.554867 29130 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44079
I20260812 18:54:40.555161 29130 heartbeater.cc:464] Registering TS with master...
I20260812 18:54:40.555814 29130 heartbeater.cc:511] Master 127.16.40.254:44079 requested a full tablet report, sending...
I20260812 18:54:40.557250 28621 ts_manager.cc:194] Registered new tserver with Master: d2885a72fc094ae39321a477b2cd2336 (127.16.40.195:34057)
I20260812 18:54:40.557924 28621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:40221
I20260812 18:54:40.562258 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:54:40.573503 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.564406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.16.40.254:44079)
0812 18:54:40.564594 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.564598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.564762 (+ 164us) server_negotiation.cc:407] Connection header received
0812 18:54:40.564871 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.564873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.564917 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.564993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:54:40.565884 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.566444 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.567550 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.567798 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.567914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.569644 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:54:40.569648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:54:40.569650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:54:40.569680 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:54:40.571283 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.572008 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.572022 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.572023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.572125 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:54:40.572452 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:54:40.572455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:54:40.572456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:54:40.572839 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:54:40.572963 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.573245 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.573372 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 18:54:40.577252 28621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:54:40.580485 28621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:54:40.596830 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.590548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34057 (local address 127.0.0.1:34460)
0812 18:54:40.591077 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.591138 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.591286 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.591992 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.591999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.592025 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:40.592214 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.592223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.593826 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.593831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.595157 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.595168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596190 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.596194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.596302 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.596341 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.596487 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.596659 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":84,"threads_started":1}
I20260812 18:54:40.596837 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.590687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.16.40.195:34057)
0812 18:54:40.591398 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.591413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.591441 (+ 28us) server_negotiation.cc:407] Connection header received
0812 18:54:40.591522 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.591527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.591788 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.592396 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:40.592586 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.593684 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.595301 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596058 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.596187 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.596348 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.596519 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.596661 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":162,"threads_started":1}
I20260812 18:54:40.597139 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.590705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43003 (local address 127.0.0.1:42272)
0812 18:54:40.591538 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.591553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.591649 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.592747 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.592753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.592777 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:40.592909 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.592916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.594418 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.594422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.595570 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.595590 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596726 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.596730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.596820 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.596843 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.596898 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.596969 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":24,"thread_start_us":128,"threads_started":1}
I20260812 18:54:40.597287 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.591326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:43699 (local address 127.0.0.1:36882)
0812 18:54:40.591649 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.591661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.591737 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.592952 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.592956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.592976 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:40.593070 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.593076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.594460 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.594464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.595595 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.595611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596907 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.596911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.596999 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.597024 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.597094 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.597157 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":25,"thread_start_us":121,"threads_started":1}
I20260812 18:54:40.597828 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.591804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.16.40.193:43003)
0812 18:54:40.592199 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.592205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.592229 (+ 24us) server_negotiation.cc:407] Connection header received
0812 18:54:40.592304 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.592309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.592504 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.592751 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:40.593057 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.594282 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.595715 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596580 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.597341 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.597522 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.597570 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.597667 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":99,"threads_started":1}
I20260812 18:54:40.597831 29148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.591855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.16.40.194:43699)
0812 18:54:40.592333 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.592347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.592371 (+ 24us) server_negotiation.cc:407] Connection header received
0812 18:54:40.592449 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.592454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.592669 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.592848 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:40.593199 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.594313 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.595771 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.596580 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.596700 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.596845 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.597573 (+ 728us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.597667 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":148,"threads_started":1}
I20260812 18:54:40.600469 29058 tablet_service.cc:1467] Processing CreateTablet for tablet aea10979f1cc4087abf7928b96e95988 (DEFAULT_TABLE table=test-table [id=c64315ada433448baa6cacfee0523a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:40.601076 28764 tablet_service.cc:1467] Processing CreateTablet for tablet aea10979f1cc4087abf7928b96e95988 (DEFAULT_TABLE table=test-table [id=c64315ada433448baa6cacfee0523a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:40.601069 28911 tablet_service.cc:1467] Processing CreateTablet for tablet aea10979f1cc4087abf7928b96e95988 (DEFAULT_TABLE table=test-table [id=c64315ada433448baa6cacfee0523a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:54:40.601795 29058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea10979f1cc4087abf7928b96e95988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:40.602104 28911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea10979f1cc4087abf7928b96e95988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:40.602105 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea10979f1cc4087abf7928b96e95988. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:54:40.609491 29149 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Bootstrap starting.
I20260812 18:54:40.610900 29151 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Bootstrap starting.
I20260812 18:54:40.610914 29150 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: Bootstrap starting.
I20260812 18:54:40.612465 29149 tablet_bootstrap.cc:654] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:40.613632 29150 tablet_bootstrap.cc:654] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:40.614002 29151 tablet_bootstrap.cc:654] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Neither blocks nor log segments found. Creating new log.
I20260812 18:54:40.614427 29149 log.cc:824] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:40.614442 29150 log.cc:824] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:40.614919 29151 log.cc:824] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Log is configured to *not* fsync() on all Append() calls
I20260812 18:54:40.616953 29149 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: No bootstrap required, opened a new log
I20260812 18:54:40.616967 29151 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: No bootstrap required, opened a new log
I20260812 18:54:40.617220 29151 ts_tablet_manager.cc:1397] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 18:54:40.617223 29149 ts_tablet_manager.cc:1397] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260812 18:54:40.618115 29150 tablet_bootstrap.cc:492] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: No bootstrap required, opened a new log
I20260812 18:54:40.618330 29150 ts_tablet_manager.cc:1397] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 18:54:40.620504 29151 raft_consensus.cc:348] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.620850 29151 raft_consensus.cc:374] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:40.620910 29151 raft_consensus.cc:729] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca84eeaf93f4eeb91ea66aa42b11724, State: Initialized, Role: FOLLOWER
I20260812 18:54:40.621443 29151 consensus_queue.cc:260] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.621402 29149 raft_consensus.cc:348] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.621779 29149 raft_consensus.cc:374] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:40.621836 29149 raft_consensus.cc:729] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2885a72fc094ae39321a477b2cd2336, State: Initialized, Role: FOLLOWER
I20260812 18:54:40.622346 29149 consensus_queue.cc:260] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.622679 29151 ts_tablet_manager.cc:1428] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 18:54:40.623026 28983 heartbeater.cc:503] Master 127.16.40.254:44079 was elected leader, sending a full tablet report...
I20260812 18:54:40.623339 29149 ts_tablet_manager.cc:1428] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:54:40.623754 29130 heartbeater.cc:503] Master 127.16.40.254:44079 was elected leader, sending a full tablet report...
I20260812 18:54:40.624079 29150 raft_consensus.cc:348] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.624384 29150 raft_consensus.cc:374] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:54:40.624449 29150 raft_consensus.cc:729] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8490783ebc674dc593ca41e7be480460, State: Initialized, Role: FOLLOWER
I20260812 18:54:40.624922 29150 consensus_queue.cc:260] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.625705 28838 heartbeater.cc:503] Master 127.16.40.254:44079 was elected leader, sending a full tablet report...
I20260812 18:54:40.625995 29150 ts_tablet_manager.cc:1428] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 18:54:40.680160 29156 raft_consensus.cc:482] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:54:40.680433 29156 raft_consensus.cc:504] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.681830 29156 leader_election.cc:288] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2885a72fc094ae39321a477b2cd2336 (127.16.40.195:34057), 8490783ebc674dc593ca41e7be480460 (127.16.40.193:43003)
I20260812 18:54:40.686630 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.682342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:57193 (local address 127.16.40.195:34057)
0812 18:54:40.682460 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.682465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.682484 (+ 19us) server_negotiation.cc:407] Connection header received
0812 18:54:40.682553 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.682561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.682609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.682680 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:40.683028 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.683729 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.685417 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.686016 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.686157 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.686216 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.686414 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.686503 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 18:54:40.686630 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.682114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:34057 (local address 127.16.40.194:57193)
0812 18:54:40.682346 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.682362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.682462 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.682722 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.682725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.682745 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:40.682872 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.682878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.683853 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.683856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.685264 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.685275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.686164 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.686167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.686274 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.686295 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.686394 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.686477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":29}
I20260812 18:54:40.687530 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.682116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43003 (local address 127.16.40.194:36249)
0812 18:54:40.683171 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.683187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:54:40.683288 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:54:40.683539 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:54:40.683542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:54:40.683560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:54:40.683681 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.683688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.684864 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.684870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.685976 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:54:40.685988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.687058 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.687062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:54:40.687176 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.687209 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 18:54:40.687301 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 18:54:40.687369 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":636,"thread_start_us":124,"threads_started":1}
I20260812 18:54:40.687548 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.682307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:36249 (local address 127.16.40.193:43003)
0812 18:54:40.682459 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.682464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.683290 (+ 826us) server_negotiation.cc:407] Connection header received
0812 18:54:40.683372 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.683375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.683429 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.683501 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:54:40.683808 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.684727 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.686147 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.686769 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.686895 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.686941 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.687328 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.687431 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 18:54:40.687605 29079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea10979f1cc4087abf7928b96e95988" candidate_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2885a72fc094ae39321a477b2cd2336" is_pre_election: true
I20260812 18:54:40.687960 29079 raft_consensus.cc:2454] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca84eeaf93f4eeb91ea66aa42b11724 in term 0.
I20260812 18:54:40.688268 28787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea10979f1cc4087abf7928b96e95988" candidate_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8490783ebc674dc593ca41e7be480460" is_pre_election: true
I20260812 18:54:40.688570 28862 leader_election.cc:302] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca84eeaf93f4eeb91ea66aa42b11724, d2885a72fc094ae39321a477b2cd2336; no voters:
I20260812 18:54:40.688616 28787 raft_consensus.cc:2454] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca84eeaf93f4eeb91ea66aa42b11724 in term 0.
I20260812 18:54:40.688900 29156 raft_consensus.cc:2781] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:54:40.689005 29156 raft_consensus.cc:482] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:54:40.689051 29156 raft_consensus.cc:3037] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:40.690393 29156 raft_consensus.cc:504] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.691107 29079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea10979f1cc4087abf7928b96e95988" candidate_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2885a72fc094ae39321a477b2cd2336"
I20260812 18:54:40.691268 29079 raft_consensus.cc:3037] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:40.691443 29156 leader_election.cc:288] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [CANDIDATE]: Term 1 election: Requested vote from peers d2885a72fc094ae39321a477b2cd2336 (127.16.40.195:34057), 8490783ebc674dc593ca41e7be480460 (127.16.40.193:43003)
I20260812 18:54:40.691855 28787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea10979f1cc4087abf7928b96e95988" candidate_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8490783ebc674dc593ca41e7be480460"
I20260812 18:54:40.692015 28787 raft_consensus.cc:3037] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:54:40.692732 29079 raft_consensus.cc:2454] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca84eeaf93f4eeb91ea66aa42b11724 in term 1.
I20260812 18:54:40.693044 28862 leader_election.cc:302] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ca84eeaf93f4eeb91ea66aa42b11724, d2885a72fc094ae39321a477b2cd2336; no voters:
I20260812 18:54:40.693548 28787 raft_consensus.cc:2454] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca84eeaf93f4eeb91ea66aa42b11724 in term 1.
I20260812 18:54:40.693807 29156 raft_consensus.cc:2781] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:54:40.694104 29156 raft_consensus.cc:686] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca84eeaf93f4eeb91ea66aa42b11724, State: Running, Role: LEADER
I20260812 18:54:40.694478 29156 consensus_queue.cc:237] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } }
I20260812 18:54:40.698148 28620 catalog_manager.cc:5495] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca84eeaf93f4eeb91ea66aa42b11724 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca84eeaf93f4eeb91ea66aa42b11724" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43699 } health_report { overall_health: HEALTHY } } }
I20260812 18:54:40.724495 29148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.720643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.16.40.194:43699)
0812 18:54:40.720801 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 18:54:40.720806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:54:40.720822 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:54:40.720913 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:54:40.720916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:54:40.720984 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:54:40.721108 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:54:40.721380 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.721898 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.722847 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:54:40.723084 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:54:40.723199 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:54:40.723564 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:54:40.724045 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:54:40.724200 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 18:54:40.724263 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 18:54:40.724330 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 18:54:40.731388 28787 raft_consensus.cc:1261] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 1 FOLLOWER]: Refusing update from remote peer 9ca84eeaf93f4eeb91ea66aa42b11724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:40.731388 29079 raft_consensus.cc:1261] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 1 FOLLOWER]: Refusing update from remote peer 9ca84eeaf93f4eeb91ea66aa42b11724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:54:40.732318 29156 consensus_queue.cc:1035] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2885a72fc094ae39321a477b2cd2336" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:40.732583 29160 consensus_queue.cc:1035] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8490783ebc674dc593ca41e7be480460" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:54:40.750731 28621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:40.750933 28621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:54:40.753854 28621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 808605f63f9d4e63a4488ea75dbccfe2: Sending DeleteTablet for 3 replicas of tablet aea10979f1cc4087abf7928b96e95988
I20260812 18:54:40.755002 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet aea10979f1cc4087abf7928b96e95988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:42272
I20260812 18:54:40.755038 28911 tablet_service.cc:1514] Processing DeleteTablet for tablet aea10979f1cc4087abf7928b96e95988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:36882
I20260812 18:54:40.755110 29058 tablet_service.cc:1514] Processing DeleteTablet for tablet aea10979f1cc4087abf7928b96e95988 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:54:40 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:34460
I20260812 18:54:40.755718 29175 tablet_replica.cc:330] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: stopping tablet replica
I20260812 18:54:40.755728 29174 tablet_replica.cc:330] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460: stopping tablet replica
I20260812 18:54:40.756126 29174 raft_consensus.cc:2229] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:40.756125 29175 raft_consensus.cc:2229] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:54:40.756229 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28693
I20260812 18:54:40.756511 29175 raft_consensus.cc:2258] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:40.756524 29174 raft_consensus.cc:2258] T aea10979f1cc4087abf7928b96e95988 P 8490783ebc674dc593ca41e7be480460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:40.756978 29173 tablet_replica.cc:330] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: stopping tablet replica
I20260812 18:54:40.757424 29173 raft_consensus.cc:2229] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:54:40.758060 29173 raft_consensus.cc:2258] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:54:40.759807 29173 ts_tablet_manager.cc:1905] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:40.760918 29175 ts_tablet_manager.cc:1905] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:54:40.763527 29173 ts_tablet_manager.cc:1918] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:40.763639 29173 log.cc:1196] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/aea10979f1cc4087abf7928b96e95988
I20260812 18:54:40.763715 29175 ts_tablet_manager.cc:1918] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:54:40.763808 29175 log.cc:1196] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/aea10979f1cc4087abf7928b96e95988
I20260812 18:54:40.763974 29173 ts_tablet_manager.cc:1939] T aea10979f1cc4087abf7928b96e95988 P 9ca84eeaf93f4eeb91ea66aa42b11724: Deleting consensus metadata
I20260812 18:54:40.764158 29175 ts_tablet_manager.cc:1939] T aea10979f1cc4087abf7928b96e95988 P d2885a72fc094ae39321a477b2cd2336: Deleting consensus metadata
I20260812 18:54:40.765640 28606 catalog_manager.cc:4841] TS d2885a72fc094ae39321a477b2cd2336 (127.16.40.195:34057): tablet aea10979f1cc4087abf7928b96e95988 (table test-table [id=c64315ada433448baa6cacfee0523a0d]) successfully deleted
I20260812 18:54:40.765911 28606 catalog_manager.cc:4841] TS 9ca84eeaf93f4eeb91ea66aa42b11724 (127.16.40.194:43699): tablet aea10979f1cc4087abf7928b96e95988 (table test-table [id=c64315ada433448baa6cacfee0523a0d]) successfully deleted
W20260812 18:54:40.771727 28605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:43003 (error 108)
I20260812 18:54:40.772709 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28843
I20260812 18:54:40.773072 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:54:40.772732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43003 (local address 127.0.0.1:42280)
0812 18:54:40.772875 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 18:54:40.772954 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 18:54:40.778366 28605 catalog_manager.cc:4584] TS 8490783ebc674dc593ca41e7be480460 (127.16.40.193:43003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aea10979f1cc4087abf7928b96e95988: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43003: connect: Connection refused (error 111)
I20260812 18:54:40.785569 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28988
I20260812 18:54:40.796099 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 28587
2026-08-12T18:54:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2622 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[29179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[29179](info): set up 2 sockets
Aug 12 18:54:40 dist-test-slave-rnrw krb5kdc[29179](info): commencing operation
krb5kdc: starting...
W20260812 18:54:41.950987 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:54:42 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:54:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangeradmin_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangeradmin_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangerlookup_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangerlookup_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
I20260812 18:54:42.153144 16547 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTWy6xo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/postgres -l logfile start
2026-08-12 18:54:45.311 UTC [29213] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 18:54:45.311 UTC [29213] LOG: listening on IPv4 address "127.16.40.212", port 49001
2026-08-12 18:54:45.316 UTC [29213] LOG: listening on Unix socket "/tmp/.s.PGSQL.49001"
2026-08-12 18:54:45.341 UTC [29216] LOG: database system was shut down at 2026-08-12 18:54:43 UTC
2026-08-12 18:54:45.346 UTC [29213] LOG: database system is ready to accept connections
I20260812 18:54:46.379504 16547 mini_postgres.cc:96] Postgres bound to 49001
2026-08-12 18:54:46.388 UTC [29225] FATAL: database "slave" does not exist
127.16.40.212:49001 - accepting connections
I20260812 18:54:46.389895 16547 mini_ranger.cc:161] Starting Ranger...
I20260812 18:54:46.410084 16547 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 18:54:47.028934 16547 mini_ranger.cc:87] Created ranger Postgres database
I20260812 18:54:47.029055 16547 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin
2026-08-12 18:54:47,369 [I] DB FLAVOR :POSTGRES
2026-08-12 18:54:47,369 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 18:54:47,369 [I] Checking connection..
2026-08-12 18:54:47,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 18:54:47,637 [I] Checking connection passed.
2026-08-12 18:54:47,638 [I] --------- Verifying version history table ---------
2026-08-12 18:54:47,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:54:47,911 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 18:54:47,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:54:48,183 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 18:54:48,183 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 18:54:48,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 18:54:48.439 UTC [29294] WARNING: there is no transaction in progress
2026-08-12 18:54:48,448 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 18:54:48,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:54:48,732 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 18:54:48,732 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 18:54:48,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 18:54:49,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 18:54:49,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rnrw.c.gcp-upstream.internal','N') ;"
2026-08-12 18:54:49,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 18:54:49,853 [I] Table x_portal_user does not exist in database ranger
2026-08-12 18:54:49,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 18:54:50,137 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 18:54:50,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 18:54:50,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 18:54:50,695 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 18:54:50,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 18:54:51.858 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:51.879 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:51.899 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:51.919 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:51.940 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:51.959 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.024 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.032 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.050 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.062 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.073 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.089 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.101 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.109 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.118 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:52.127 UTC [29420] WARNING: there is no transaction in progress
2026-08-12 18:54:53,005 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 18:54:53,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 18:54:53,284 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 18:54:53,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 18:54:53,565 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 18:54:53,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 18:54:53,829 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 18:54:53,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 18:54:54,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 18:54:54,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rnrw.c.gcp-upstream.internal';"
2026-08-12 18:54:54,617 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 18:54:54,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 18:54:54,897 [I] DB_PATCHES have already been applied
I20260812 18:54:54.903865 16547 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTWy6xo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 18:54:54.903962 16547 mini_ranger.cc:193] Using host: 127.16.40.212
I20260812 18:54:54.909125 16547 mini_ranger.cc:227] Ranger admin URL: http://127.16.40.212:39119
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 6:54:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 18:54:56.140941 16547 mini_ranger.cc:160] Time spent starting Ranger: real 9.751s user 0.000s sys 0.014s
Aug 12, 2026 6:55:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 6:55:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 6:55:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 6:55:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 6:55:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 18:55:20.841202 16547 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangerkms_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/rangerkms_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.40.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 18:55:20 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 18:55:20.945775 16547 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 18:55:20.958498 16547 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 18:55:22.010874 16547 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 18:55:22.011027 16547 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms
I20260812 18:55:22.011139 16547 mini_ranger_kms.cc:227] Using postgres at 127.16.40.212:49001
2026-08-12 18:55:22,282 [I] DB FLAVOR :POSTGRES
2026-08-12 18:55:22,282 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 18:55:22,283 [I] Checking connection
2026-08-12 18:55:22,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 18:55:22,549 [I] connection success
2026-08-12 18:55:22,549 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 18:55:22,549 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 18:55:22,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 18:55:22,835 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 18:55:22,835 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 18:55:22,836 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 18:55:22,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:49001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 18:55:23,110 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 18:55:23.666225 16547 mini_ranger_kms.cc:310] Created kms service
I20260812 18:55:23.865022 16547 mini_ranger_kms.cc:326] Created kudu user
I20260812 18:55:23.919903 16547 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 18:55:23 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:24.387285 16547 mini_ranger_kms.cc:384] Added ranger policy
I20260812 18:55:24.387481 16547 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 18:55:24.387521 16547 mini_ranger_kms.cc:240] Using host: 127.16.40.212
I20260812 18:55:24.393430 16547 mini_ranger_kms.cc:276] Ranger KMS PID: 29622
I20260812 18:55:24.393618 16547 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.40.212:43205
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 6:55:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 18:55:26 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786560926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:55:26.942359 16547 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.997s user 0.015s sys 0.240s
I20260812 18:55:26.942498 16547 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 18:55:26.942618 16547 mini_ranger_kms.cc:401] 127.16.40.212:43205/kms/v1/keys
Aug 12 18:55:27 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
Aug 12 18:55:27 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
Aug 12 18:55:29 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:55:29.989567 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:42967
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:42967
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:43205/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:55:30.121170 29680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:55:30.121487 29680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:55:30.121531 29680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:55:30.126093 29680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:55:30.126186 29680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:55:30.126206 29680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:55:30.126223 29680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:55:30.126240 29680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:55:30.132359 29680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.40.212:43205/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:42967
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:55:30.133633 29680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:55:30.134675 29680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:55:30.144420 29685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:30.144465 29680 server_base.cc:1034] running on GCE node
W20260812 18:55:30.144407 29686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:30.144407 29688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:30.145370 29680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:55:30.146596 29680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:55:30.147770 29680 hybrid_clock.cc:648] HybridClock initialized: now 1786560930147750 us; error 34 us; skew 500 ppm
Aug 12 18:55:30 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560930, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:55:30.154825 29680 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:55:30.155969 29680 webserver.cc:460] Webserver started at http://127.16.40.254:34643/ using document root <none> and password file <none>
I20260812 18:55:30.156692 29680 fs_manager.cc:359] Metadata directory not provided
I20260812 18:55:30.156788 29680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:55:30.157033 29680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:55:30 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560930, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:30.217705 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68"
format_stamp: "Formatted at 2026-08-12 18:55:30 on dist-test-slave-rnrw"
server_key: "5348677d42096f0350f9d17ed4378dd0"
server_key_iv: "ba8f67891401a3e911d4e710dd1f04ce"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:30.218570 29680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68"
format_stamp: "Formatted at 2026-08-12 18:55:30 on dist-test-slave-rnrw"
server_key: "5348677d42096f0350f9d17ed4378dd0"
server_key_iv: "ba8f67891401a3e911d4e710dd1f04ce"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:30.224603 29680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 18:55:30.244673 29695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:30.246583 29680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:55:30.246796 29680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68"
format_stamp: "Formatted at 2026-08-12 18:55:30 on dist-test-slave-rnrw"
server_key: "5348677d42096f0350f9d17ed4378dd0"
server_key_iv: "ba8f67891401a3e911d4e710dd1f04ce"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:30.246964 29680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:55:30.274530 29680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:55:30.278740 29680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:55:30.278995 29680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:55:30.288627 29680 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:42967
I20260812 18:55:30.288668 29753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:42967 every 8 connection(s)
I20260812 18:55:30.290103 29680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:55:30.290395 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 29680
I20260812 18:55:30.290612 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:55:30.290959 16547 external_mini_cluster.cc:1428] Setting key 79624d57682345297ad3fb54fe1da7fa
I20260812 18:55:30.294112 29754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 18:55:30 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:55:30.302383 29754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: Bootstrap starting.
I20260812 18:55:30.305446 29754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: Neither blocks nor log segments found. Creating new log.
I20260812 18:55:30.306923 29754 log.cc:824] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: Log is configured to *not* fsync() on all Append() calls
I20260812 18:55:30.307026 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:30.292456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.16.40.254:42967)
0812 18:55:30.293081 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:30.293097 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:30.293434 (+ 337us) server_negotiation.cc:407] Connection header received
0812 18:55:30.294354 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:30.294373 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:30.294740 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:30.295111 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:55:30.295991 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:30.297175 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:30.298153 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:30.298653 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:30.298834 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:30.301613 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:55:30.301619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:55:30.301635 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:55:30.301669 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:55:30.303739 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:30.304266 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:30.304272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:30.304278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:30.304392 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:30.304789 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:30.304793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:30.304795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:30.305244 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:55:30.305469 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:30.305817 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 18:55:30.306263 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":148,"threads_started":1}
I20260812 18:55:30.309566 29754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: No bootstrap required, opened a new log
I20260812 18:55:30.313912 29754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } }
I20260812 18:55:30.314263 29754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:55:30.314317 29754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076f79ff2bfc4a6cb22b7a8eed9aac68, State: Initialized, Role: FOLLOWER
I20260812 18:55:30.314821 29754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } }
I20260812 18:55:30.315008 29754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:55:30.315083 29754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:55:30.315187 29754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:55:30.316457 29754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } }
I20260812 18:55:30.316906 29754 leader_election.cc:302] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 076f79ff2bfc4a6cb22b7a8eed9aac68; no voters:
I20260812 18:55:30.317318 29754 leader_election.cc:288] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:55:30.317487 29759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:55:30.318570 29759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [term 1 LEADER]: Becoming Leader. State: Replica: 076f79ff2bfc4a6cb22b7a8eed9aac68, State: Running, Role: LEADER
I20260812 18:55:30.318775 29754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:55:30.319130 29759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } }
I20260812 18:55:30.320847 29760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } } }
I20260812 18:55:30.320972 29760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [sys.catalog]: This master's current role is: LEADER
I20260812 18:55:30.321215 29760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 076f79ff2bfc4a6cb22b7a8eed9aac68. Latest consensus state: current_term: 1 leader_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076f79ff2bfc4a6cb22b7a8eed9aac68" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42967 } } }
I20260812 18:55:30.321290 29760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68 [sys.catalog]: This master's current role is: LEADER
I20260812 18:55:30.321300 29754 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 18:55:30.325636 29774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:55:30.331131 29774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:55:30.342221 29774 catalog_manager.cc:1347] Generated new cluster ID: ace1a30159854df5a82f281933724abc
I20260812 18:55:30.342303 29774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:55:30.390848 29774 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:55:30.392388 29774 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:55:30.401557 29774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: Generated new TSK 0
I20260812 18:55:30.402657 29774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:55:31.221127 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:43205/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:55:31.407742 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:55:31.408237 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:55:31.408370 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:55:31.414036 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:55:31.414144 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:55:31.414258 29802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:55:31.420719 29802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.40.212:43205/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:55:31.422014 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:55:31.423132 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:55:31.434767 29812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:31.434767 29809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:31.434924 29810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:31.435730 29802 server_base.cc:1034] running on GCE node
I20260812 18:55:31.436340 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:55:31.437287 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:55:31.440675 29802 hybrid_clock.cc:648] HybridClock initialized: now 1786560931440626 us; error 56 us; skew 500 ppm
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:55:31.444384 29802 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:55:31.445735 29802 webserver.cc:460] Webserver started at http://127.16.40.193:46251/ using document root <none> and password file <none>
I20260812 18:55:31.446421 29802 fs_manager.cc:359] Metadata directory not provided
I20260812 18:55:31.446539 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:55:31.446821 29802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:31.468142 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "bf2841643ab4468e8012afabc8c7d5a4"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "1c9f332ccfa9057c919182c82267bddb"
server_key_iv: "c84047cfeae7a0407f59fc44724f8bdc"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.469380 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2841643ab4468e8012afabc8c7d5a4"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "1c9f332ccfa9057c919182c82267bddb"
server_key_iv: "c84047cfeae7a0407f59fc44724f8bdc"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.475945 29802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 18:55:31.499400 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:31.501540 29802 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260812 18:55:31.501735 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "bf2841643ab4468e8012afabc8c7d5a4"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "1c9f332ccfa9057c919182c82267bddb"
server_key_iv: "c84047cfeae7a0407f59fc44724f8bdc"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.501906 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:55:31.521272 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:55:31.529702 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:55:31.530107 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:55:31.531112 29802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:55:31.532475 29802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:55:31.532667 29802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 18:55:31.532883 29802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:55:31.533007 29802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:31.555871 29802 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:34927
I20260812 18:55:31.557479 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:55:31.559854 29944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:34927 every 8 connection(s)
I20260812 18:55:31.560230 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 29802
I20260812 18:55:31.560384 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:55:31.560775 16547 external_mini_cluster.cc:1428] Setting key 36b51906e5832f56bbbba8e2084d97f1
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:55:31.586267 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:31.565932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:45493 (local address 127.16.40.254:42967)
0812 18:55:31.566330 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:31.566336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:31.568603 (+ 2267us) server_negotiation.cc:407] Connection header received
0812 18:55:31.568861 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:31.568865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:31.568933 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:31.569040 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:55:31.570950 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.571831 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.575795 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.576208 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.576307 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:31.580396 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:55:31.580401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:55:31.580404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:55:31.580440 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:55:31.582669 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:31.583784 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:31.583788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:31.583790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:31.583958 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:31.584694 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:31.584698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:31.584700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:31.584972 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:55:31.585065 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:31.585917 (+ 852us) server_negotiation.cc:299] Negotiation successful
0812 18:55:31.586072 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":164,"threads_started":1}
I20260812 18:55:31.588671 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:31.564839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42967 (local address 127.16.40.193:45493)
0812 18:55:31.566712 (+ 1873us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:31.566781 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:31.568367 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:31.569394 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:31.569404 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:31.569988 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:55:31.570754 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:31.570775 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.572004 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.572020 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:31.573600 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:31.573613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.576599 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.576604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:31.576906 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:31.577780 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:55:31.577821 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:55:31.580089 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:55:31.582988 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:31.583003 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:31.583048 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:31.583642 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:31.584212 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:31.584215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:31.584225 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:31.584571 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:31.585144 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:55:31.585152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:55:31.585533 (+ 381us) client_negotiation.cc:769] Sending connection context
0812 18:55:31.587006 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0812 18:55:31.587469 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:55:31.590574 29946 heartbeater.cc:347] Connected to a master server at 127.16.40.254:42967
I20260812 18:55:31.590969 29946 heartbeater.cc:464] Registering TS with master...
I20260812 18:55:31.591853 29946 heartbeater.cc:511] Master 127.16.40.254:42967 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:55:31.594357 29715 ts_manager.cc:194] Registered new tserver with Master: bf2841643ab4468e8012afabc8c7d5a4 (127.16.40.193:34927)
I20260812 18:55:31.596185 29715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:45493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:55:31.621203 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:43205/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 18:55:31.793373 29953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:55:31.793764 29953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:55:31.793870 29953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:55:31.798380 29953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:55:31.798489 29953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:55:31.798585 29953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:55:31.805079 29953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.40.212:43205/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:55:31.806326 29953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:55:31.807468 29953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:55:31.819356 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:31.819356 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:31.819556 29953 server_base.cc:1034] running on GCE node
W20260812 18:55:31.819762 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:31.820356 29953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:55:31.821296 29953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:55:31.822494 29953 hybrid_clock.cc:648] HybridClock initialized: now 1786560931822477 us; error 46 us; skew 500 ppm
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:55:31.825784 29953 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:55:31.827040 29953 webserver.cc:460] Webserver started at http://127.16.40.194:46711/ using document root <none> and password file <none>
I20260812 18:55:31.827776 29953 fs_manager.cc:359] Metadata directory not provided
I20260812 18:55:31.827879 29953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:55:31.828202 29953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:31.857466 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "ec216f2778db4129823228dc874ad27a"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "03084f7da92f1a7a036c92a100b990e8"
server_key_iv: "72df164bc170d60016c0af81fb72270e"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.858350 29953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "ec216f2778db4129823228dc874ad27a"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "03084f7da92f1a7a036c92a100b990e8"
server_key_iv: "72df164bc170d60016c0af81fb72270e"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.864962 29953 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 18:55:31.897207 29969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:31.899492 29953 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260812 18:55:31.899796 29953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "ec216f2778db4129823228dc874ad27a"
format_stamp: "Formatted at 2026-08-12 18:55:31 on dist-test-slave-rnrw"
server_key: "03084f7da92f1a7a036c92a100b990e8"
server_key_iv: "72df164bc170d60016c0af81fb72270e"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:31.900104 29953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:55:31.922639 29953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:55:31.927675 29953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:55:31.928049 29953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:55:31.929041 29953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:55:31.930356 29953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:55:31.930487 29953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:31.930606 29953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:55:31.930704 29953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:31.956642 29953 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:39443
I20260812 18:55:31.956918 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:39443 every 8 connection(s)
I20260812 18:55:31.958479 29953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:55:31.963387 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 29953
I20260812 18:55:31.963541 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:55:31.963905 16547 external_mini_cluster.cc:1428] Setting key 29226557830530502946b88b2a93bac2
Aug 12 18:55:31 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:55:31.984575 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:31.964560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:42413 (local address 127.16.40.254:42967)
0812 18:55:31.964735 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:31.964741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:31.965442 (+ 701us) server_negotiation.cc:407] Connection header received
0812 18:55:31.967410 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:31.967414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:31.967494 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:31.967580 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:55:31.970241 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.971184 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.973578 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.974011 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.974149 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:31.979079 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:55:31.979083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:55:31.979086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:55:31.979122 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:55:31.981110 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:31.982040 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:31.982045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:31.982047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:31.982264 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:31.982846 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:31.982850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:31.982852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:31.983132 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:55:31.983234 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:31.984102 (+ 868us) server_negotiation.cc:299] Negotiation successful
0812 18:55:31.984270 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:55:31.986542 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:31.962730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42967 (local address 127.16.40.194:42413)
0812 18:55:31.965205 (+ 2475us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:31.965298 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:31.966962 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:31.968178 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:31.968190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:31.968858 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:55:31.969969 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:31.970005 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.971434 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.971439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:31.973375 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:31.973393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:31.974434 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:31.974599 (+ 165us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:31.974927 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:31.976947 (+ 2020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:55:31.976974 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:55:31.978792 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:55:31.981333 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:31.981342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:31.981365 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:31.981894 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:31.982448 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:31.982452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:31.982454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:31.982696 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:31.983324 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:55:31.983333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:55:31.983756 (+ 423us) client_negotiation.cc:769] Sending connection context
0812 18:55:31.984881 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0812 18:55:31.985327 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":316,"threads_started":1}
I20260812 18:55:31.988651 30093 heartbeater.cc:347] Connected to a master server at 127.16.40.254:42967
I20260812 18:55:31.988989 30093 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:55:31.989729 30093 heartbeater.cc:511] Master 127.16.40.254:42967 requested a full tablet report, sending...
I20260812 18:55:31.991185 29715 ts_manager.cc:194] Registered new tserver with Master: ec216f2778db4129823228dc874ad27a (127.16.40.194:39443)
I20260812 18:55:31.992064 29715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:42413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:55:32.012773 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:43205/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:55:32.218976 30099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:55:32.219445 30099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:55:32.219579 30099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:55:32.227545 30099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:55:32.227650 30099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:55:32.227756 30099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:55:32.234798 30099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.40.212:43205/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:42967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:55:32.236509 30099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:55:32.238134 30099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:55:32.259037 30105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:32.259843 30108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:55:32.260767 30106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:55:32.261700 30099 server_base.cc:1034] running on GCE node
I20260812 18:55:32.262578 30099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:55:32.263551 30099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 12 18:55:32 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:32.266852 30099 hybrid_clock.cc:648] HybridClock initialized: now 1786560932266755 us; error 104 us; skew 500 ppm
Aug 12 18:55:32 dist-test-slave-rnrw krb5kdc[29179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560932, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:55:32.272960 30099 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:55:32.274391 30099 webserver.cc:460] Webserver started at http://127.16.40.195:39839/ using document root <none> and password file <none>
I20260812 18:55:32.275290 30099 fs_manager.cc:359] Metadata directory not provided
I20260812 18:55:32.275466 30099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:55:32.275825 30099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:55:32 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560932, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:55:32.311044 30099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "19c6eb6007374317afe2899fb82085ba"
format_stamp: "Formatted at 2026-08-12 18:55:32 on dist-test-slave-rnrw"
server_key: "bb059cdd0f93645b80e0ef15988ddcea"
server_key_iv: "50b88ef30526f7217da7b37a6b2c81d4"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:32.312048 30099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "19c6eb6007374317afe2899fb82085ba"
format_stamp: "Formatted at 2026-08-12 18:55:32 on dist-test-slave-rnrw"
server_key: "bb059cdd0f93645b80e0ef15988ddcea"
server_key_iv: "50b88ef30526f7217da7b37a6b2c81d4"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:32.318078 30099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 18:55:32.341190 30116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:32.343129 30099 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260812 18:55:32.343290 30099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "19c6eb6007374317afe2899fb82085ba"
format_stamp: "Formatted at 2026-08-12 18:55:32 on dist-test-slave-rnrw"
server_key: "bb059cdd0f93645b80e0ef15988ddcea"
server_key_iv: "50b88ef30526f7217da7b37a6b2c81d4"
server_key_version: "kuduclusterkey@0"
I20260812 18:55:32.343432 30099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:55:32.377593 30099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:55:32.383246 30099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:55:32.383585 30099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:55:32.384652 30099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:55:32.385946 30099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:55:32.386058 30099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:32.386173 30099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:55:32.386237 30099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:55:32.411904 30099 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:45885
I20260812 18:55:32.413380 30099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:55:32.421752 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 30099
I20260812 18:55:32.421891 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:55:32.422199 16547 external_mini_cluster.cc:1428] Setting key 912fb6f725b94e71aacac53fb2a7f6c0
I20260812 18:55:32.415678 30236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:45885 every 8 connection(s)
Aug 12 18:55:32 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560932, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:55:32.443151 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.420266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:35285 (local address 127.16.40.254:42967)
0812 18:55:32.420404 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.420409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.421378 (+ 969us) server_negotiation.cc:407] Connection header received
0812 18:55:32.426307 (+ 4929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.426312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.426374 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.426427 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:55:32.428725 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.429594 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.431858 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.432229 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.432312 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.438192 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:55:32.438196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:55:32.438199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:55:32.438233 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:55:32.440216 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:32.441054 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:32.441058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:32.441060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:32.441162 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:32.441673 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:32.441677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:32.441678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:32.441926 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:55:32.442011 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.442807 (+ 796us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.442968 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 18:55:32.445487 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.418746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42967 (local address 127.16.40.195:35285)
0812 18:55:32.421183 (+ 2437us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.421248 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.426002 (+ 4754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.426932 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.426944 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.427580 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:55:32.428477 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.428499 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.430011 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.430015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.431673 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.431688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.432616 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.432621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.432936 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.433748 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:55:32.433771 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:55:32.437941 (+ 4170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:55:32.440391 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:32.440401 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:32.440426 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:32.440916 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:32.441276 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:55:32.441279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:55:32.441281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:55:32.441490 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:55:32.442094 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:55:32.442101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:55:32.442485 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.443702 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.444145 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2187,"thread_start_us":309,"threads_started":1}
I20260812 18:55:32.447494 30237 heartbeater.cc:347] Connected to a master server at 127.16.40.254:42967
I20260812 18:55:32.447791 30237 heartbeater.cc:464] Registering TS with master...
I20260812 18:55:32.448472 30237 heartbeater.cc:511] Master 127.16.40.254:42967 requested a full tablet report, sending...
I20260812 18:55:32.451550 29715 ts_manager.cc:194] Registered new tserver with Master: 19c6eb6007374317afe2899fb82085ba (127.16.40.195:45885)
I20260812 18:55:32.452194 29715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:35285
I20260812 18:55:32.463505 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:55:32.474212 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.465970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.16.40.254:42967)
0812 18:55:32.466112 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.466117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.466440 (+ 323us) server_negotiation.cc:407] Connection header received
0812 18:55:32.466545 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.466548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.466605 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.466697 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:55:32.467463 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.468009 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.468932 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.469171 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.469292 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.470796 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:55:32.470800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:55:32.470803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:55:32.470839 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:55:32.472366 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:32.472865 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:32.472869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:32.472871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:32.472963 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:55:32.473265 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:55:32.473269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:55:32.473271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:55:32.473474 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:55:32.473624 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.473895 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.474021 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 18:55:32.478474 29715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:55:32.483600 29715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:55:32.523729 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.506071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.16.40.193:34927)
0812 18:55:32.508963 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.508972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.511807 (+ 2835us) server_negotiation.cc:407] Connection header received
0812 18:55:32.511974 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.511982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.512351 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.512568 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:55:32.517167 (+ 4599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.518653 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.520806 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.522234 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.522340 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.522542 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.523305 (+ 763us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.523497 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2728,"thread_start_us":145,"threads_started":1}
I20260812 18:55:32.524863 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.505892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34927 (local address 127.0.0.1:48562)
0812 18:55:32.511672 (+ 5780us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.511692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.511799 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.516658 (+ 4859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.516663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.516702 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:55:32.516926 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.516938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.518929 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.518933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.520516 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.520533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.522711 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.522716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.522987 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.523059 (+ 72us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.524415 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.524685 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5692,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260812 18:55:32.525810 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.500933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:45885 (local address 127.0.0.1:51338)
0812 18:55:32.508060 (+ 7127us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.508138 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.508311 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.513070 (+ 4759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.513073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.513107 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:55:32.513306 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.513323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.517008 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.517012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.518680 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.518700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.525409 (+ 6709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.525412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.525510 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.525535 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.525598 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.525678 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6826,"thread_start_us":153,"threads_started":1}
I20260812 18:55:32.529273 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.504418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39443 (local address 127.0.0.1:38478)
0812 18:55:32.508590 (+ 4172us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.508605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.508689 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.511134 (+ 2445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.511142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.511174 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:55:32.511370 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.511378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.522605 (+ 11227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.522609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.525228 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.525245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.528811 (+ 3566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.528816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.528917 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.528945 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.529021 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.529115 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4084,"mutex_wait_us":1330,"thread_start_us":214,"threads_started":1}
I20260812 18:55:32.530136 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.502379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51338 (local address 127.16.40.195:45885)
0812 18:55:32.512180 (+ 9801us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.512205 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.512232 (+ 27us) server_negotiation.cc:407] Connection header received
0812 18:55:32.512310 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.512316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.512583 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.512772 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:55:32.515526 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.516831 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.522927 (+ 6096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.523837 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.523956 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.524132 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.529839 (+ 5707us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.529980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9647,"thread_start_us":195,"threads_started":1}
I20260812 18:55:32.529691 29871 tablet_service.cc:1467] Processing CreateTablet for tablet 64abae70d7474516a8259696701c53d0 (DEFAULT_TABLE table=test-table [id=ccbc618cc437459ebd5dece20a1872f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:55:32.530675 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.504988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.16.40.194:39443)
0812 18:55:32.509097 (+ 4109us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.509105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.509146 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:55:32.509287 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.509294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.509649 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.513576 (+ 3927us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:55:32.513916 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.522433 (+ 8517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.527553 (+ 5120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.528430 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.528573 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.528733 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.530413 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.530537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3892,"thread_start_us":96,"threads_started":1}
I20260812 18:55:32.531452 29871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64abae70d7474516a8259696701c53d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:55:32.534353 30162 tablet_service.cc:1467] Processing CreateTablet for tablet 64abae70d7474516a8259696701c53d0 (DEFAULT_TABLE table=test-table [id=ccbc618cc437459ebd5dece20a1872f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:55:32.537408 30162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64abae70d7474516a8259696701c53d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:55:32.551136 30020 tablet_service.cc:1467] Processing CreateTablet for tablet 64abae70d7474516a8259696701c53d0 (DEFAULT_TABLE table=test-table [id=ccbc618cc437459ebd5dece20a1872f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:55:32.552820 30020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64abae70d7474516a8259696701c53d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:55:32.565057 30260 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Bootstrap starting.
I20260812 18:55:32.568778 30260 tablet_bootstrap.cc:654] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Neither blocks nor log segments found. Creating new log.
I20260812 18:55:32.569849 30260 log.cc:824] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Log is configured to *not* fsync() on all Append() calls
I20260812 18:55:32.572005 30262 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Bootstrap starting.
I20260812 18:55:32.575811 30262 tablet_bootstrap.cc:654] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Neither blocks nor log segments found. Creating new log.
I20260812 18:55:32.576648 30263 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Bootstrap starting.
I20260812 18:55:32.580318 30263 tablet_bootstrap.cc:654] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Neither blocks nor log segments found. Creating new log.
I20260812 18:55:32.581861 30262 log.cc:824] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Log is configured to *not* fsync() on all Append() calls
I20260812 18:55:32.583106 30260 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: No bootstrap required, opened a new log
I20260812 18:55:32.583478 30260 ts_tablet_manager.cc:1397] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.007s
I20260812 18:55:32.583760 30263 log.cc:824] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Log is configured to *not* fsync() on all Append() calls
I20260812 18:55:32.588364 30262 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: No bootstrap required, opened a new log
I20260812 18:55:32.588768 30262 ts_tablet_manager.cc:1397] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260812 18:55:32.589535 30260 raft_consensus.cc:348] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.590063 30260 raft_consensus.cc:374] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:55:32.590191 30260 raft_consensus.cc:729] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2841643ab4468e8012afabc8c7d5a4, State: Initialized, Role: FOLLOWER
I20260812 18:55:32.590910 30260 consensus_queue.cc:260] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.592371 30260 ts_tablet_manager.cc:1428] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260812 18:55:32.592931 29946 heartbeater.cc:503] Master 127.16.40.254:42967 was elected leader, sending a full tablet report...
I20260812 18:55:32.594585 30262 raft_consensus.cc:348] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.595041 30262 raft_consensus.cc:374] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:55:32.595139 30262 raft_consensus.cc:729] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec216f2778db4129823228dc874ad27a, State: Initialized, Role: FOLLOWER
I20260812 18:55:32.595722 30262 consensus_queue.cc:260] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.598824 30263 tablet_bootstrap.cc:492] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: No bootstrap required, opened a new log
I20260812 18:55:32.599135 30263 ts_tablet_manager.cc:1397] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20260812 18:55:32.597189 30262 ts_tablet_manager.cc:1428] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260812 18:55:32.607009 30093 heartbeater.cc:503] Master 127.16.40.254:42967 was elected leader, sending a full tablet report...
I20260812 18:55:32.606354 30263 raft_consensus.cc:348] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.607583 30263 raft_consensus.cc:374] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:55:32.607746 30263 raft_consensus.cc:729] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c6eb6007374317afe2899fb82085ba, State: Initialized, Role: FOLLOWER
I20260812 18:55:32.608420 30263 consensus_queue.cc:260] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.609848 30263 ts_tablet_manager.cc:1428] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260812 18:55:32.610342 30237 heartbeater.cc:503] Master 127.16.40.254:42967 was elected leader, sending a full tablet report...
W20260812 18:55:32.671198 30238 tablet.cc:2369] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 18:55:32 dist-test-slave-rnrw krb5kdc[29179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560931, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
W20260812 18:55:32.710558 30094 tablet.cc:2369] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:55:32.731858 30270 raft_consensus.cc:482] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:55:32.732210 30270 raft_consensus.cc:504] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.733716 30270 leader_election.cc:288] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec216f2778db4129823228dc874ad27a (127.16.40.194:39443), bf2841643ab4468e8012afabc8c7d5a4 (127.16.40.193:34927)
I20260812 18:55:32.740058 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.734147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:39443 (local address 127.16.40.195:54203)
0812 18:55:32.735042 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.735059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.735155 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.735816 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.735819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.735838 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:55:32.736052 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.736059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.737132 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.737136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.738374 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.738386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.739549 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.739556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.739705 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.739732 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.739812 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.739890 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810}
I20260812 18:55:32.740468 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.735428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:54203 (local address 127.16.40.194:39443)
0812 18:55:32.735545 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.735550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.735565 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:55:32.735622 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.735627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.735684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.735763 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:55:32.736194 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.736993 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.738544 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.739236 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.739387 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.739468 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.740270 (+ 802us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.740350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 18:55:32.741552 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abae70d7474516a8259696701c53d0" candidate_uuid: "19c6eb6007374317afe2899fb82085ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec216f2778db4129823228dc874ad27a" is_pre_election: true
I20260812 18:55:32.741930 30046 raft_consensus.cc:2454] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19c6eb6007374317afe2899fb82085ba in term 0.
I20260812 18:55:32.742663 30120 leader_election.cc:302] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c6eb6007374317afe2899fb82085ba, ec216f2778db4129823228dc874ad27a; no voters:
I20260812 18:55:32.743011 30270 raft_consensus.cc:2781] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:55:32.743116 30270 raft_consensus.cc:482] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:55:32.743158 30270 raft_consensus.cc:3037] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:55:32.744880 30270 raft_consensus.cc:504] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.745299 30270 leader_election.cc:288] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [CANDIDATE]: Term 1 election: Requested vote from peers ec216f2778db4129823228dc874ad27a (127.16.40.194:39443), bf2841643ab4468e8012afabc8c7d5a4 (127.16.40.193:34927)
I20260812 18:55:32.746690 30046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abae70d7474516a8259696701c53d0" candidate_uuid: "19c6eb6007374317afe2899fb82085ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec216f2778db4129823228dc874ad27a"
I20260812 18:55:32.746837 30046 raft_consensus.cc:3037] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:55:32.748593 30046 raft_consensus.cc:2454] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c6eb6007374317afe2899fb82085ba in term 1.
I20260812 18:55:32.749046 30120 leader_election.cc:302] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19c6eb6007374317afe2899fb82085ba, ec216f2778db4129823228dc874ad27a; no voters:
I20260812 18:55:32.749246 30270 raft_consensus.cc:2781] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:55:32.749504 30270 raft_consensus.cc:686] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 1 LEADER]: Becoming Leader. State: Replica: 19c6eb6007374317afe2899fb82085ba, State: Running, Role: LEADER
I20260812 18:55:32.749922 30270 consensus_queue.cc:237] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } }
I20260812 18:55:32.758380 29713 catalog_manager.cc:5495] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 19c6eb6007374317afe2899fb82085ba (127.16.40.195). New cstate: current_term: 1 leader_uuid: "19c6eb6007374317afe2899fb82085ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19c6eb6007374317afe2899fb82085ba" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 } health_report { overall_health: UNKNOWN } } }
I20260812 18:55:32.761310 30259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.735577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:45341 (local address 127.16.40.193:34927)
0812 18:55:32.735757 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.735761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.738866 (+ 3105us) server_negotiation.cc:407] Connection header received
0812 18:55:32.739069 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.739072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.739122 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.739168 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:55:32.739803 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.740535 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.759619 (+ 19084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.760275 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.760443 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.760506 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.761054 (+ 548us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.761159 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 18:55:32.762151 30273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.734480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34927 (local address 127.16.40.195:45341)
0812 18:55:32.738750 (+ 4270us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.738767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:55:32.738952 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:55:32.739297 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:55:32.739299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:55:32.739315 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:55:32.739663 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.739671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.742428 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.742432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.757591 (+ 15159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:55:32.757610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.760658 (+ 3048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.760665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:55:32.760827 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.760859 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 18:55:32.761668 (+ 809us) client_negotiation.cc:240] Negotiation successful
0812 18:55:32.761776 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4194,"thread_start_us":191,"threads_started":1}
I20260812 18:55:32.763155 29884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abae70d7474516a8259696701c53d0" candidate_uuid: "19c6eb6007374317afe2899fb82085ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2841643ab4468e8012afabc8c7d5a4"
I20260812 18:55:32.763221 29883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64abae70d7474516a8259696701c53d0" candidate_uuid: "19c6eb6007374317afe2899fb82085ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2841643ab4468e8012afabc8c7d5a4" is_pre_election: true
I20260812 18:55:32.763410 29884 raft_consensus.cc:3037] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:55:32.766431 29884 raft_consensus.cc:2454] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19c6eb6007374317afe2899fb82085ba in term 1.
W20260812 18:55:32.814139 29947 tablet.cc:2369] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:55:32.823377 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.817720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.16.40.195:45885)
0812 18:55:32.817847 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 18:55:32.817853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:55:32.817868 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:55:32.817971 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:55:32.817974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:55:32.818031 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:55:32.818165 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:55:32.818505 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.819353 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.820840 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:55:32.821186 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:55:32.821295 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:55:32.821838 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:55:32.822483 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:55:32.822613 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 18:55:32.823094 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 18:55:32.823211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 18:55:32.831498 29884 raft_consensus.cc:1261] T 64abae70d7474516a8259696701c53d0 P bf2841643ab4468e8012afabc8c7d5a4 [term 1 FOLLOWER]: Refusing update from remote peer 19c6eb6007374317afe2899fb82085ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:55:32.831593 30046 raft_consensus.cc:1261] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 1 FOLLOWER]: Refusing update from remote peer 19c6eb6007374317afe2899fb82085ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:55:32.832301 30274 consensus_queue.cc:1035] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2841643ab4468e8012afabc8c7d5a4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:55:32.832762 30270 consensus_queue.cc:1035] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec216f2778db4129823228dc874ad27a" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:55:32.853745 30277 mvcc.cc:205] Tried to move back new op lower bound from 7317753580868321280 to 7317753580546473984. Current Snapshot: MvccSnapshot[applied={T|T < 7317753580868321280}]
I20260812 18:55:32.864014 30278 mvcc.cc:205] Tried to move back new op lower bound from 7317753580868321280 to 7317753580546473984. Current Snapshot: MvccSnapshot[applied={T|T < 7317753580868321280}]
I20260812 18:55:32.875025 29715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:55:32.875237 29715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:55:32.878515 29715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 076f79ff2bfc4a6cb22b7a8eed9aac68: Sending DeleteTablet for 3 replicas of tablet 64abae70d7474516a8259696701c53d0
I20260812 18:55:32.879808 30162 tablet_service.cc:1514] Processing DeleteTablet for tablet 64abae70d7474516a8259696701c53d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:55:32 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:51338
I20260812 18:55:32.881508 30020 tablet_service.cc:1514] Processing DeleteTablet for tablet 64abae70d7474516a8259696701c53d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:55:32 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:38478
I20260812 18:55:32.882230 29871 tablet_service.cc:1514] Processing DeleteTablet for tablet 64abae70d7474516a8259696701c53d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:55:32 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:48562
I20260812 18:55:32.883936 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 29802
I20260812 18:55:32.884804 30289 tablet_replica.cc:330] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: stopping tablet replica
I20260812 18:55:32.885183 30289 raft_consensus.cc:2229] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:55:32.885582 30289 raft_consensus.cc:2258] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:55:32.892915 30290 tablet_replica.cc:330] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: stopping tablet replica
I20260812 18:55:32.893402 30290 raft_consensus.cc:2229] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:55:32.893919 30290 raft_consensus.cc:2258] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:55:32.895867 30289 ts_tablet_manager.cc:1905] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:55:32.899521 30289 ts_tablet_manager.cc:1918] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:55:32.899667 30289 log.cc:1196] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/64abae70d7474516a8259696701c53d0
I20260812 18:55:32.900008 30289 ts_tablet_manager.cc:1939] T 64abae70d7474516a8259696701c53d0 P 19c6eb6007374317afe2899fb82085ba: Deleting consensus metadata
I20260812 18:55:32.907068 30290 ts_tablet_manager.cc:1905] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:55:32.910691 30290 ts_tablet_manager.cc:1918] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:55:32.910846 30290 log.cc:1196] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/64abae70d7474516a8259696701c53d0
I20260812 18:55:32.911468 30290 ts_tablet_manager.cc:1939] T 64abae70d7474516a8259696701c53d0 P ec216f2778db4129823228dc874ad27a: Deleting consensus metadata
I20260812 18:55:32.919248 29698 catalog_manager.cc:4841] TS 19c6eb6007374317afe2899fb82085ba (127.16.40.195:45885): tablet 64abae70d7474516a8259696701c53d0 (table test-table [id=ccbc618cc437459ebd5dece20a1872f1]) successfully deleted
W20260812 18:55:32.920128 29700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:34927 (error 108)
I20260812 18:55:32.920948 29700 catalog_manager.cc:4841] TS ec216f2778db4129823228dc874ad27a (127.16.40.194:39443): tablet 64abae70d7474516a8259696701c53d0 (table test-table [id=ccbc618cc437459ebd5dece20a1872f1]) successfully deleted
I20260812 18:55:32.921353 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 29953
I20260812 18:55:32.939097 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:55:32.938650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34927 (local address 127.0.0.1:48576)
0812 18:55:32.938887 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 18:55:32.938984 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260812 18:55:32.939486 29700 catalog_manager.cc:4584] TS bf2841643ab4468e8012afabc8c7d5a4 (127.16.40.193:34927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64abae70d7474516a8259696701c53d0: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34927: connect: Connection refused (error 111)
I20260812 18:55:32.953172 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 30099
I20260812 18:55:32.988934 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 29680
I20260812 18:55:33.006253 16547 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 18:55:33.352308 16547 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 18:55:33.352470 16547 mini_ranger.cc:66] Stopping Ranger...
I20260812 18:55:33.699573 16547 mini_ranger.cc:68] Stopped Ranger
2026-08-12 18:55:33.699 UTC [29213] LOG: received smart shutdown request
2026-08-12 18:55:33.705 UTC [29213] LOG: background worker "logical replication launcher" (PID 29223) exited with exit code 1
2026-08-12 18:55:33.705 UTC [29217] LOG: shutting down
2026-08-12 18:55:33.888 UTC [29213] LOG: database system is shut down
2026-08-12T18:55:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53209 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:55:34 dist-test-slave-rnrw krb5kdc[30305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:55:34 dist-test-slave-rnrw krb5kdc[30305](info): set up 2 sockets
Aug 12 18:55:34 dist-test-slave-rnrw krb5kdc[30305](info): commencing operation
krb5kdc: starting...
W20260812 18:55:35.168783 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:55:35 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:55:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangeradmin_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangeradmin_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangerlookup_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangerlookup_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
I20260812 18:55:35.384640 16547 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTWy6xo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/postgres -l logfile start
2026-08-12 18:55:38.389 UTC [30339] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 18:55:38.389 UTC [30339] LOG: listening on IPv4 address "127.16.40.212", port 46185
2026-08-12 18:55:38.394 UTC [30339] LOG: listening on Unix socket "/tmp/.s.PGSQL.46185"
2026-08-12 18:55:38.418 UTC [30342] LOG: database system was shut down at 2026-08-12 18:55:36 UTC
2026-08-12 18:55:38.423 UTC [30339] LOG: database system is ready to accept connections
I20260812 18:55:39.463021 16547 mini_postgres.cc:96] Postgres bound to 46185
2026-08-12 18:55:39.471 UTC [30350] FATAL: database "slave" does not exist
127.16.40.212:46185 - accepting connections
I20260812 18:55:39.472246 16547 mini_ranger.cc:161] Starting Ranger...
I20260812 18:55:39.492513 16547 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 18:55:40.170426 16547 mini_ranger.cc:87] Created ranger Postgres database
I20260812 18:55:40.170528 16547 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin
2026-08-12 18:55:40,474 [I] DB FLAVOR :POSTGRES
2026-08-12 18:55:40,474 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 18:55:40,475 [I] Checking connection..
2026-08-12 18:55:40,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 18:55:40,765 [I] Checking connection passed.
2026-08-12 18:55:40,765 [I] --------- Verifying version history table ---------
2026-08-12 18:55:40,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:55:41,042 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 18:55:41,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:55:41,324 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 18:55:41,324 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 18:55:41,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 18:55:41.587 UTC [30424] WARNING: there is no transaction in progress
2026-08-12 18:55:41,596 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 18:55:41,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 18:55:41,855 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 18:55:41,856 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 18:55:41,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 18:55:42,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 18:55:42,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rnrw.c.gcp-upstream.internal','N') ;"
2026-08-12 18:55:42,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 18:55:42,961 [I] Table x_portal_user does not exist in database ranger
2026-08-12 18:55:42,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 18:55:43,253 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 18:55:43,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 18:55:43,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 18:55:43,790 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 18:55:43,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 18:55:45.048 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.071 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.095 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.115 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.139 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.161 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.231 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.241 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.261 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.274 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.287 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.305 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.318 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.328 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.338 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:45.349 UTC [30543] WARNING: there is no transaction in progress
2026-08-12 18:55:46,259 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 18:55:46,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 18:55:46,523 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 18:55:46,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 18:55:46,819 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 18:55:46,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 18:55:47,124 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 18:55:47,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 18:55:47,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 18:55:47,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rnrw.c.gcp-upstream.internal';"
2026-08-12 18:55:47,955 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 18:55:47,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 18:55:48,218 [I] DB_PATCHES have already been applied
I20260812 18:55:48.224804 16547 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTWy6xo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 18:55:48.224889 16547 mini_ranger.cc:193] Using host: 127.16.40.212
I20260812 18:55:48.230322 16547 mini_ranger.cc:227] Ranger admin URL: http://127.16.40.212:44889
Aug 12, 2026 6:55:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 6:55:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 6:55:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 6:55:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 6:55:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 6:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 6:55:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 18:55:49.500999 16547 mini_ranger.cc:160] Time spent starting Ranger: real 10.029s user 0.001s sys 0.013s
Aug 12, 2026 6:56:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 6:56:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 6:56:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 6:56:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 6:56:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 18:56:14.622912 16547 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.40.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.40.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangerkms_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.40.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/rangerkms_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.40.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.40.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.40.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/HTTP_127.16.40.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 18:56:14 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 18:56:14.727312 16547 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 18:56:14.741339 16547 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 18:56:15.705741 16547 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 18:56:15.705844 16547 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms
I20260812 18:56:15.705865 16547 mini_ranger_kms.cc:227] Using postgres at 127.16.40.212:46185
2026-08-12 18:56:15,902 [I] DB FLAVOR :POSTGRES
2026-08-12 18:56:15,903 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 18:56:15,903 [I] Checking connection
2026-08-12 18:56:15,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 18:56:16,165 [I] connection success
2026-08-12 18:56:16,165 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 18:56:16,165 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 18:56:16,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 18:56:16,423 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 18:56:16,423 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 18:56:16,423 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 18:56:16,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.40.212:46185/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 18:56:16,702 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 18:56:17.281555 16547 mini_ranger_kms.cc:310] Created kms service
I20260812 18:56:17.501497 16547 mini_ranger_kms.cc:326] Created kudu user
I20260812 18:56:17.553676 16547 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 18:56:17 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560974, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:17.968828 16547 mini_ranger_kms.cc:384] Added ranger policy
I20260812 18:56:17.968994 16547 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTWy6xo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTWy6xo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 18:56:17.969033 16547 mini_ranger_kms.cc:240] Using host: 127.16.40.212
I20260812 18:56:17.973673 16547 mini_ranger_kms.cc:276] Ranger KMS PID: 30745
I20260812 18:56:17.973773 16547 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.40.212:46555
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 6:56:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 18:56:20 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786560980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:20.545588 16547 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.818s user 0.018s sys 0.164s
I20260812 18:56:20.545732 16547 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 18:56:20.545817 16547 mini_ranger_kms.cc:401] 127.16.40.212:46555/kms/v1/keys
Aug 12 18:56:20 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
Aug 12 18:56:21 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560980, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.40.212@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
Aug 12 18:56:23 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:56:23.943651 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38745
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38745
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:24.093907 30803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:24.094244 30803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:24.094285 30803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:24.098908 30803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:24.099010 30803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:24.099030 30803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:24.099047 30803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:24.099063 30803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:24.105330 30803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.40.212:46555/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38745
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:24.106591 30803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:24.107738 30803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:24.117590 30808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:24.117676 30809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:24.117882 30803 server_base.cc:1034] running on GCE node
W20260812 18:56:24.117671 30812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:24.118692 30803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:24.120105 30803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:24.121335 30803 hybrid_clock.cc:648] HybridClock initialized: now 1786560984121314 us; error 42 us; skew 500 ppm
Aug 12 18:56:24 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560984, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:24.130275 30803 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:24.131722 30803 webserver.cc:460] Webserver started at http://127.16.40.254:40913/ using document root <none> and password file <none>
I20260812 18:56:24.132344 30803 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:24.132434 30803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:24.132751 30803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:56:24 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560984, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:24.183866 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "5426513e0b9f47a2bc6207220a4d4184"
format_stamp: "Formatted at 2026-08-12 18:56:24 on dist-test-slave-rnrw"
server_key: "9f0805270eaee3b84608aa11ac11191f"
server_key_iv: "718097a03e691e98b64feee33d27e2f3"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:24.184626 30803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "5426513e0b9f47a2bc6207220a4d4184"
format_stamp: "Formatted at 2026-08-12 18:56:24 on dist-test-slave-rnrw"
server_key: "9f0805270eaee3b84608aa11ac11191f"
server_key_iv: "718097a03e691e98b64feee33d27e2f3"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:24.189003 30803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:56:24.206164 30819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:24.207842 30803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:56:24.208021 30803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "5426513e0b9f47a2bc6207220a4d4184"
format_stamp: "Formatted at 2026-08-12 18:56:24 on dist-test-slave-rnrw"
server_key: "9f0805270eaee3b84608aa11ac11191f"
server_key_iv: "718097a03e691e98b64feee33d27e2f3"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:24.208189 30803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:24.223826 30803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:24.227670 30803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:24.227937 30803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:24.237682 30803 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38745
I20260812 18:56:24.237703 30874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38745 every 8 connection(s)
I20260812 18:56:24.238843 30803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:24.242309 30875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:24.244396 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 30803
I20260812 18:56:24.244586 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:56:24.244948 16547 external_mini_cluster.cc:1428] Setting key b5222f0d2484c9926c22803b863b3335
I20260812 18:56:24.250799 30875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Bootstrap starting.
I20260812 18:56:24.253911 30875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:24.255008 30875 log.cc:824] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:56:24 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:24.257802 30875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: No bootstrap required, opened a new log
I20260812 18:56:24.263322 30875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:24.263757 30875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:24.263841 30875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5426513e0b9f47a2bc6207220a4d4184, State: Initialized, Role: FOLLOWER
I20260812 18:56:24.264417 30875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:24.264623 30875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:24.264737 30875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:24.264743 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:24.246742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.16.40.254:38745)
0812 18:56:24.247351 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:24.247374 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:24.247444 (+ 70us) server_negotiation.cc:407] Connection header received
0812 18:56:24.248427 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:24.248465 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:24.249026 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:24.249459 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:24.250552 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:24.252218 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:24.253599 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:24.254363 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:24.254565 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:24.258221 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:24.258230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:24.258255 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:24.258299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:24.260700 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:24.261368 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:24.261376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:24.261383 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:24.261552 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:24.261942 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:24.261946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:24.261948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:24.262663 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:24.262915 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:24.263251 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 18:56:24.263585 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":205,"threads_started":1}
I20260812 18:56:24.264880 30875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:24.266260 30875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:24.266681 30875 leader_election.cc:302] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5426513e0b9f47a2bc6207220a4d4184; no voters:
I20260812 18:56:24.267066 30875 leader_election.cc:288] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:56:24.267251 30881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:24.267854 30881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 LEADER]: Becoming Leader. State: Replica: 5426513e0b9f47a2bc6207220a4d4184, State: Running, Role: LEADER
I20260812 18:56:24.268272 30881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:24.268808 30875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:24.269822 30881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5426513e0b9f47a2bc6207220a4d4184. Latest consensus state: current_term: 1 leader_uuid: "5426513e0b9f47a2bc6207220a4d4184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } } }
I20260812 18:56:24.269987 30881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:24.270270 30884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5426513e0b9f47a2bc6207220a4d4184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } } }
I20260812 18:56:24.270387 30884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:24.271577 30875 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 18:56:24.273351 30893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:24.281438 30893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:24.290038 30893 catalog_manager.cc:1347] Generated new cluster ID: 50bd702919d74b378f56900906fe937f
I20260812 18:56:24.290127 30893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:24.334815 30893 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:56:24.336535 30893 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:24.347975 30893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Generated new TSK 0
I20260812 18:56:24.349222 30893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:56:25.104115 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:25.248773 30929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:25.249140 30929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:25.249202 30929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:25.254428 30929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:25.254539 30929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:25.254635 30929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:25.261402 30929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:25.262744 30929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:25.263903 30929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:25.275221 30939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:25.275779 30929 server_base.cc:1034] running on GCE node
W20260812 18:56:25.275779 30934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:25.276225 30935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:25.276849 30929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:25.277782 30929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:25.279007 30929 hybrid_clock.cc:648] HybridClock initialized: now 1786560985278979 us; error 42 us; skew 500 ppm
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:25.283357 30929 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:25.284713 30929 webserver.cc:460] Webserver started at http://127.16.40.193:45165/ using document root <none> and password file <none>
I20260812 18:56:25.285403 30929 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:25.285519 30929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:25.285805 30929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:25.305946 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "45b74a4c420e46a0b1692e35574c4c8e"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "b5583eefbd02724ebf0b5705d7fe6684"
server_key_iv: "4c748c015391161c822e3ee0d8e02de7"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.307163 30929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "45b74a4c420e46a0b1692e35574c4c8e"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "b5583eefbd02724ebf0b5705d7fe6684"
server_key_iv: "4c748c015391161c822e3ee0d8e02de7"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.313341 30929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 18:56:25.329845 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.331549 30929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:25.331743 30929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "45b74a4c420e46a0b1692e35574c4c8e"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "b5583eefbd02724ebf0b5705d7fe6684"
server_key_iv: "4c748c015391161c822e3ee0d8e02de7"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.331911 30929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:25.342877 30929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:25.346486 30929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:25.346738 30929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:25.347512 30929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:25.348753 30929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:25.348830 30929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.348924 30929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:25.348981 30929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.361512 30929 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:33801
I20260812 18:56:25.362999 30929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:25.367998 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 30929
I20260812 18:56:25.368165 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:56:25.368527 16547 external_mini_cluster.cc:1428] Setting key 9f7214c59728586495217d2ffdd44cae
I20260812 18:56:25.387436 31075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:33801 every 8 connection(s)
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:56:25.415036 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:25.387938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:52663 (local address 127.16.40.254:38745)
0812 18:56:25.398694 (+ 10756us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:25.398698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:25.398711 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:56:25.398751 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:25.398753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:25.398809 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:25.398931 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:25.400608 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.401570 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.404077 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.404416 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.404524 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:25.407958 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:25.407964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:25.407967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:25.408017 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:25.410179 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:25.410926 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:25.410930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:25.410932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:25.411053 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:25.411860 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:25.411864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:25.411866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:25.412147 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:25.412257 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:25.414624 (+ 2367us) server_negotiation.cc:299] Negotiation successful
0812 18:56:25.414815 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10665,"thread_start_us":122,"threads_started":1}
I20260812 18:56:25.416903 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:25.388523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.193:52663)
0812 18:56:25.397068 (+ 8545us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:25.397175 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:25.398255 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:25.399180 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:25.399190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:25.399590 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:25.400348 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:25.400371 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.401751 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.401755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:25.403895 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:25.403910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.404761 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.404766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:25.404985 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:25.405757 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:25.405780 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:25.407709 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:25.410364 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:25.410370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:25.410394 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:25.410790 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:25.411171 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:25.411175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:25.411177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:25.411731 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:25.413418 (+ 1687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:25.413428 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:25.413998 (+ 570us) client_negotiation.cc:769] Sending connection context
0812 18:56:25.414392 (+ 394us) client_negotiation.cc:240] Negotiation successful
0812 18:56:25.415597 (+ 1205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8129,"thread_start_us":166,"threads_started":1}
I20260812 18:56:25.418715 31076 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:25.418999 31076 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:25.419694 31076 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:25.421937 30838 ts_manager.cc:194] Registered new tserver with Master: 45b74a4c420e46a0b1692e35574c4c8e (127.16.40.193:33801)
I20260812 18:56:25.423745 30838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:52663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:56:25.452520 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 18:56:25.627899 31084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:25.628382 31084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:25.628458 31084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:25.635155 31084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:25.635390 31084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:25.635521 31084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:25.644248 31084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:25.645814 31084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:25.647245 31084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:25.661523 31091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:25.662145 31093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:25.665256 31090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:25.666061 31084 server_base.cc:1034] running on GCE node
I20260812 18:56:25.666847 31084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:25.668015 31084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:25.669243 31084 hybrid_clock.cc:648] HybridClock initialized: now 1786560985669204 us; error 60 us; skew 500 ppm
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:25.673763 31084 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:25.675197 31084 webserver.cc:460] Webserver started at http://127.16.40.194:45549/ using document root <none> and password file <none>
I20260812 18:56:25.676079 31084 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:25.676193 31084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:25.676537 31084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:25.699723 31084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "f5d6d2fdce5f4d6aa737674ea919100f"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "03099d1221dbbf27a71081e9f165af98"
server_key_iv: "9962fdd38ead843267cf42ee16d0a142"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.700732 31084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "f5d6d2fdce5f4d6aa737674ea919100f"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "03099d1221dbbf27a71081e9f165af98"
server_key_iv: "9962fdd38ead843267cf42ee16d0a142"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.711107 31084 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260812 18:56:25.737118 31101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.741683 31084 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.001s
I20260812 18:56:25.741882 31084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "f5d6d2fdce5f4d6aa737674ea919100f"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "03099d1221dbbf27a71081e9f165af98"
server_key_iv: "9962fdd38ead843267cf42ee16d0a142"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:25.742082 31084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:25.768399 31084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:25.772738 31084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:25.772974 31084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:25.773785 31084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:25.775029 31084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:25.775101 31084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.775166 31084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:25.775194 31084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:25.787350 31084 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35533
I20260812 18:56:25.787812 31228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35533 every 8 connection(s)
I20260812 18:56:25.788722 31084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:25.795544 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31084
I20260812 18:56:25.795727 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:56:25.796125 16547 external_mini_cluster.cc:1428] Setting key 2923b7380bf1950d8d3aabc3db4f85b2
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Aug 12 18:56:25 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:25.836158 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:25.791261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:48507 (local address 127.16.40.254:38745)
0812 18:56:25.791421 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:25.791426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:25.793088 (+ 1662us) server_negotiation.cc:407] Connection header received
0812 18:56:25.794565 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:25.794570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:25.794650 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:25.794778 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:25.796616 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.797424 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.799677 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.800048 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.800148 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:25.830945 (+ 30797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:25.830951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:25.830954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:25.830992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:25.832900 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:25.833847 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:25.833851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:25.833853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:25.833959 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:25.834612 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:25.834616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:25.834618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:25.834876 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:25.834956 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:25.835803 (+ 847us) server_negotiation.cc:299] Negotiation successful
0812 18:56:25.835960 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:56:25.837984 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:25.792012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.194:48507)
0812 18:56:25.792862 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:25.792944 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:25.794238 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:25.795020 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:25.795030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:25.795491 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:25.796322 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:25.796346 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.797649 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.797655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:25.799438 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:25.799454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:25.800413 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:25.800417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:25.824885 (+ 24468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:25.828530 (+ 3645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:25.828589 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:25.830627 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:25.833178 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:25.833187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:25.833210 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:25.833690 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:25.834118 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:25.834122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:25.834124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:25.834470 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:25.835062 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:25.835070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:25.835468 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 18:56:25.836370 (+ 902us) client_negotiation.cc:240] Negotiation successful
0812 18:56:25.836796 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":216,"threads_started":1}
I20260812 18:56:25.840442 31230 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:25.840874 31230 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:25.841593 31230 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:25.842999 30838 ts_manager.cc:194] Registered new tserver with Master: f5d6d2fdce5f4d6aa737674ea919100f (127.16.40.194:35533)
I20260812 18:56:25.843827 30838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:48507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:56:25.866247 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560985, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
W20260812 18:56:26.047348 31237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:26.047910 31237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:26.048028 31237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:26.054795 31237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:26.055001 31237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:26.055151 31237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:26.063588 31237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:26.065448 31237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:26.066977 31237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:26.089711 31242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:26.089711 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:26.092820 31237 server_base.cc:1034] running on GCE node
W20260812 18:56:26.092713 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:26.094028 31237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:26.095328 31237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:26.098845 31237 hybrid_clock.cc:648] HybridClock initialized: now 1786560986098714 us; error 138 us; skew 500 ppm
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560986, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:26.106570 31237 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:26.108527 31237 webserver.cc:460] Webserver started at http://127.16.40.195:46663/ using document root <none> and password file <none>
I20260812 18:56:26.109846 31237 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:26.110069 31237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:26.110512 31237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560986, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:26.150847 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "1edf4fb1c248481fabd649732b73317f"
format_stamp: "Formatted at 2026-08-12 18:56:26 on dist-test-slave-rnrw"
server_key: "ab600718a8f97983b325433cb4a69b23"
server_key_iv: "aab6e3e1756f6e5278d1396ed9f1e58a"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:26.152287 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "1edf4fb1c248481fabd649732b73317f"
format_stamp: "Formatted at 2026-08-12 18:56:26 on dist-test-slave-rnrw"
server_key: "ab600718a8f97983b325433cb4a69b23"
server_key_iv: "aab6e3e1756f6e5278d1396ed9f1e58a"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:26.160979 31237 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260812 18:56:26.202916 31253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:26.205672 31237 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260812 18:56:26.205875 31237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "1edf4fb1c248481fabd649732b73317f"
format_stamp: "Formatted at 2026-08-12 18:56:26 on dist-test-slave-rnrw"
server_key: "ab600718a8f97983b325433cb4a69b23"
server_key_iv: "aab6e3e1756f6e5278d1396ed9f1e58a"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:26.206054 31237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:26.244714 31237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:26.251183 31237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:26.251758 31237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:26.253057 31237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:26.254922 31237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:26.255088 31237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:26.255227 31237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:26.255308 31237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:26.308282 31237 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:44335
I20260812 18:56:26.311168 31237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:26.314374 31379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:44335 every 8 connection(s)
I20260812 18:56:26.315076 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31237
I20260812 18:56:26.315385 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:56:26.317795 16547 external_mini_cluster.cc:1428] Setting key 814a2d3282d353a9990f69169e8cb109
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560986, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:26.361522 31079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:26.325214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:36741 (local address 127.16.40.254:38745)
0812 18:56:26.325520 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:26.325525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:26.326679 (+ 1154us) server_negotiation.cc:407] Connection header received
0812 18:56:26.330521 (+ 3842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:26.330527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:26.330633 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:26.330785 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:26.335716 (+ 4931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:26.337216 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:26.341122 (+ 3906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:26.341891 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:26.342171 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:26.349957 (+ 7786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:26.349964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:26.349967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:26.350073 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:26.354495 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:26.356504 (+ 2009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:26.356512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:26.356514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:26.356801 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:26.357948 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:26.357952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:26.357954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:26.358660 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:26.358875 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:26.360745 (+ 1870us) server_negotiation.cc:299] Negotiation successful
0812 18:56:26.361147 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260812 18:56:26.366111 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:26.322721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.195:36741)
0812 18:56:26.326277 (+ 3556us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:26.326448 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:26.329799 (+ 3351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:26.331821 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:26.331852 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:26.333238 (+ 1386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:26.335313 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:26.335375 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:26.337713 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:26.337717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:26.340763 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:26.340782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:26.342430 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:26.342446 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:26.342983 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:26.344693 (+ 1710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:26.344783 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:26.349257 (+ 4474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:26.354914 (+ 5657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:26.354957 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:26.355039 (+ 82us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:26.356150 (+ 1111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:26.357038 (+ 888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:26.357060 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:26.357072 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:26.357627 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:26.358964 (+ 1337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:26.358974 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:26.359840 (+ 866us) client_negotiation.cc:769] Sending connection context
0812 18:56:26.360358 (+ 518us) client_negotiation.cc:240] Negotiation successful
0812 18:56:26.362611 (+ 2253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2909,"thread_start_us":414,"threads_started":1}
I20260812 18:56:26.370070 31380 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:26.370741 31380 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:26.372298 31380 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:26.375345 30838 ts_manager.cc:194] Registered new tserver with Master: 1edf4fb1c248481fabd649732b73317f (127.16.40.195:44335)
I20260812 18:56:26.376996 30838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:36741
I20260812 18:56:26.378507 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:56:26.380199 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 30929
I20260812 18:56:26.408679 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31084
I20260812 18:56:26.445649 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31237
I20260812 18:56:26.469693 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 30803
I20260812 18:56:26.502388 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38745
--webserver_interface=127.16.40.254
--webserver_port=40913
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38745
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:26.755126 31383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:26.755651 31383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:26.755738 31383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:26.763191 31383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:26.763370 31383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:26.763435 31383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:26.763463 31383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:26.763499 31383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:26.774435 31383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38745
--ranger_config_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=40913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:26.776897 31383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:26.779114 31383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:26.799326 31390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:26.799360 31393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:26.799335 31391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:26.800788 31383 server_base.cc:1034] running on GCE node
I20260812 18:56:26.801925 31383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:26.804157 31383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:26.807593 31383 hybrid_clock.cc:648] HybridClock initialized: now 1786560986807520 us; error 92 us; skew 500 ppm
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560986, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:26.812844 31383 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:26.814814 31383 webserver.cc:460] Webserver started at http://127.16.40.254:40913/ using document root <none> and password file <none>
I20260812 18:56:26.815994 31383 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:26.816190 31383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 18:56:26 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560986, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:26.869354 31383 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:56:26.872033 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:26.873596 31383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:56:26.873776 31383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "5426513e0b9f47a2bc6207220a4d4184"
format_stamp: "Formatted at 2026-08-12 18:56:24 on dist-test-slave-rnrw"
server_key: "9f0805270eaee3b84608aa11ac11191f"
server_key_iv: "718097a03e691e98b64feee33d27e2f3"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:26.874378 31383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:26.912657 31383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:26.916350 31383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:26.916926 31383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:26.925165 31462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38745 every 8 connection(s)
I20260812 18:56:26.925160 31383 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38745
I20260812 18:56:26.926496 31383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:26.931561 31463 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:56:26.933597 31463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Bootstrap starting.
I20260812 18:56:26.933876 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31383
I20260812 18:56:26.934427 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:33801
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:38745
--webserver_port=45165
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:26.952277 31463 log.cc:824] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:26.960088 31463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:56:26.960634 31463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Bootstrap complete.
I20260812 18:56:26.965420 31463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:26.965875 31463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5426513e0b9f47a2bc6207220a4d4184, State: Initialized, Role: FOLLOWER
I20260812 18:56:26.966490 31463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:26.966615 31463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:26.966727 31463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:26.966835 31463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:56:26.969049 31463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:26.969415 31463 leader_election.cc:302] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5426513e0b9f47a2bc6207220a4d4184; no voters:
I20260812 18:56:26.969822 31463 leader_election.cc:288] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 18:56:26.969918 31467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 18:56:26.970113 31467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [term 2 LEADER]: Becoming Leader. State: Replica: 5426513e0b9f47a2bc6207220a4d4184, State: Running, Role: LEADER
I20260812 18:56:26.970427 31467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } }
I20260812 18:56:26.971190 31463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:26.971421 31470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5426513e0b9f47a2bc6207220a4d4184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } } }
I20260812 18:56:26.971551 31470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:26.972209 31471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5426513e0b9f47a2bc6207220a4d4184. Latest consensus state: current_term: 2 leader_uuid: "5426513e0b9f47a2bc6207220a4d4184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5426513e0b9f47a2bc6207220a4d4184" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38745 } } }
I20260812 18:56:26.972321 31471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:26.972941 31473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:26.976536 31473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:26.977187 31463 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 18:56:26.977530 31473 catalog_manager.cc:1259] Loaded cluster ID: 50bd702919d74b378f56900906fe937f
I20260812 18:56:26.977648 31473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:26.982919 31473 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:26.984328 31473 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Loaded TSK: 0
I20260812 18:56:26.986131 31473 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:56:27.088635 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:27.089066 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:27.089133 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:27.095454 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:27.095551 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:27.095651 31465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:27.103907 31465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:33801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=45165
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:27.105454 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:27.106884 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:56:27.120719 31465 server_base.cc:1034] running on GCE node
W20260812 18:56:27.120774 31509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.121140 31507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.120774 31506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:27.121886 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:27.122757 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:27.123963 31465 hybrid_clock.cc:648] HybridClock initialized: now 1786560987123924 us; error 59 us; skew 500 ppm
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:27.128155 31465 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:27.129657 31465 webserver.cc:460] Webserver started at http://127.16.40.193:45165/ using document root <none> and password file <none>
I20260812 18:56:27.130497 31465 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:27.130664 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:27.166466 31465 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.000s
I20260812 18:56:27.177966 31518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.180229 31465 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260812 18:56:27.180398 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "45b74a4c420e46a0b1692e35574c4c8e"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "b5583eefbd02724ebf0b5705d7fe6684"
server_key_iv: "4c748c015391161c822e3ee0d8e02de7"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:27.181241 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:27.209239 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:27.212900 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:27.213153 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:27.213860 31465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:27.214985 31465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:27.215075 31465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.215155 31465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:27.215193 31465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.226073 31465 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:33801
I20260812 18:56:27.227571 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:27.230418 31644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:33801 every 8 connection(s)
I20260812 18:56:27.236267 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31465
I20260812 18:56:27.237021 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:35533
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:38745
--webserver_port=45549
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:27.253840 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.234181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:44691 (local address 127.16.40.254:38745)
0812 18:56:27.234891 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:27.234913 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:27.235294 (+ 381us) server_negotiation.cc:407] Connection header received
0812 18:56:27.237077 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:27.237096 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:27.237592 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:27.238008 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:27.240077 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.241742 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.243216 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.243737 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.243943 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.246960 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:27.246968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:27.246990 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:27.247032 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:27.249454 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.250287 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.250295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.250301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.250476 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.250964 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.250967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.250969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.251529 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:27.251685 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:27.252672 (+ 987us) server_negotiation.cc:299] Negotiation successful
0812 18:56:27.252985 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":179,"threads_started":1}
I20260812 18:56:27.253855 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.232395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.193:44691)
0812 18:56:27.235040 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:27.235116 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:27.236338 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:27.238568 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:27.238677 (+ 109us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:27.239076 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:27.239828 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.239854 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.241940 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.241948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.243055 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.243070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.243940 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.243945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.244146 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.244887 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:27.244907 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:27.246669 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:27.249658 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.249666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.249685 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.250115 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.250624 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.250627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.250630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.250834 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.251743 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:27.251750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:27.252093 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 18:56:27.252328 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 18:56:27.252762 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2364,"thread_start_us":179,"threads_started":1}
I20260812 18:56:27.257200 31645 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:27.257504 31645 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.258212 31645 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
W20260812 18:56:27.261096 31645 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:38745 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 18:56:27.261266 31645 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.261518 31645 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:27.262765 31645 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.263017 31645 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
W20260812 18:56:27.264014 31645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:56:27.384949 31650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:27.385370 31650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:27.385449 31650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:27.390158 31650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:27.390264 31650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:27.390368 31650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:27.397179 31650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:35533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=45549
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:27.398744 31650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:27.400134 31650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:27.414525 31660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.414505 31656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.414505 31657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:27.415855 31650 server_base.cc:1034] running on GCE node
I20260812 18:56:27.416424 31650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:27.417416 31650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:27.420804 31650 hybrid_clock.cc:648] HybridClock initialized: now 1786560987420759 us; error 68 us; skew 500 ppm
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:27.424010 31650 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:27.425211 31650 webserver.cc:460] Webserver started at http://127.16.40.194:45549/ using document root <none> and password file <none>
I20260812 18:56:27.425843 31650 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:27.425956 31650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:27.448364 31650 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 18:56:27.452028 31671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.453908 31650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260812 18:56:27.454077 31650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "f5d6d2fdce5f4d6aa737674ea919100f"
format_stamp: "Formatted at 2026-08-12 18:56:25 on dist-test-slave-rnrw"
server_key: "03099d1221dbbf27a71081e9f165af98"
server_key_iv: "9962fdd38ead843267cf42ee16d0a142"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:27.454896 31650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:27.484634 31650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:27.488675 31650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:27.488910 31650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:27.489642 31650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:27.491011 31650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:27.491088 31650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.491153 31650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:27.491187 31650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.503791 31650 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35533
I20260812 18:56:27.504180 31801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35533 every 8 connection(s)
I20260812 18:56:27.505246 31650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:27.511091 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31650
I20260812 18:56:27.511593 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.40.212:46555/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:44335
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:38745
--webserver_port=46663
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:27.524593 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.507488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:55041 (local address 127.16.40.254:38745)
0812 18:56:27.507672 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:27.507677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:27.508674 (+ 997us) server_negotiation.cc:407] Connection header received
0812 18:56:27.510007 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:27.510010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:27.510075 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:27.510156 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:27.511924 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.512759 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.514632 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.514991 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.515102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.519272 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:27.519308 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:27.519313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:27.519353 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:27.521502 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.522389 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.522396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.522398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.522511 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.523013 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.523016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.523018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.523280 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:27.523362 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:27.524201 (+ 839us) server_negotiation.cc:299] Negotiation successful
0812 18:56:27.524366 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 18:56:27.526544 31804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.507874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.194:55041)
0812 18:56:27.508445 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:27.508519 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:27.509723 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:27.510416 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:27.510427 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:27.510842 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:27.511682 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.511706 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.512973 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.512977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.514440 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.514453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.515368 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.515373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.515614 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.516658 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:27.516683 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:27.518661 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:27.521698 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.521707 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.521728 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.522220 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.522653 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.522656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.522658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.522874 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.523456 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:27.523464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:27.523871 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 18:56:27.524877 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0812 18:56:27.525309 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260812 18:56:27.528156 31802 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:27.528465 31802 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.529161 31802 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
W20260812 18:56:27.531116 31802 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:38745 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 18:56:27.531286 31802 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.531512 31802 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:27.532696 31802 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.532915 31802 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
W20260812 18:56:27.533946 31802 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:56:27.657279 31806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:27.657703 31806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:27.657774 31806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:27.663784 31806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:27.663856 31806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:27.663959 31806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:27.670555 31806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.40.212:46555/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:44335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=46663
--tserver_master_addrs=127.16.40.254:38745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:27.671960 31806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:27.673252 31806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:27.686782 31812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.686942 31813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:27.687165 31815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:27.688637 31806 server_base.cc:1034] running on GCE node
I20260812 18:56:27.689148 31806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:27.690001 31806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:27.691198 31806 hybrid_clock.cc:648] HybridClock initialized: now 1786560987691137 us; error 78 us; skew 500 ppm
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:27.694811 31806 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:27.695942 31806 webserver.cc:460] Webserver started at http://127.16.40.195:46663/ using document root <none> and password file <none>
I20260812 18:56:27.696626 31806 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:27.696709 31806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:27.724076 31806 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 18:56:27.727607 31823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.731922 31806 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260812 18:56:27.732209 31806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "1edf4fb1c248481fabd649732b73317f"
format_stamp: "Formatted at 2026-08-12 18:56:26 on dist-test-slave-rnrw"
server_key: "ab600718a8f97983b325433cb4a69b23"
server_key_iv: "aab6e3e1756f6e5278d1396ed9f1e58a"
server_key_version: "kuduclusterkey@0"
I20260812 18:56:27.733153 31806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:27.757236 31806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:27.760675 31806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:27.760888 31806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:27.761611 31806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:27.762712 31806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:27.762789 31806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.762858 31806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:27.762892 31806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:27.774331 31806 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:44335
I20260812 18:56:27.774349 31951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:44335 every 8 connection(s)
I20260812 18:56:27.775763 31806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:27.783707 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 31806
Aug 12 18:56:27 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560987, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:27.793474 31649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.777930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:40687 (local address 127.16.40.254:38745)
0812 18:56:27.779522 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:27.779527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:27.779549 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:56:27.780662 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:27.780666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:27.780720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:27.780820 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:27.782427 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.783302 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.785324 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.785707 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.785836 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.789085 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:27.789091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:27.789093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:27.789129 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:27.790958 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.791532 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.791535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.791537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.791622 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.792151 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.792157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.792161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.792432 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:27.792617 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:27.793180 (+ 563us) server_negotiation.cc:299] Negotiation successful
0812 18:56:27.793305 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478}
I20260812 18:56:27.794368 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.778422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38745 (local address 127.16.40.195:40687)
0812 18:56:27.779058 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:27.779128 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:27.780356 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:27.781032 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:27.781042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:27.781460 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:27.782223 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.782244 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.783506 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.783510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.785170 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:27.785182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.786056 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.786060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:27.786292 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.787097 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:27.787120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:27.788866 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:27.791060 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.791067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.791086 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.791438 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.791858 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:27.791862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:27.791864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:27.792037 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:27.792632 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:27.792641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:27.792920 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 18:56:27.793118 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 18:56:27.793473 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":153,"threads_started":1}
I20260812 18:56:27.795701 31952 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38745
I20260812 18:56:27.795935 31952 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:27.796615 31952 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:27.796763 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:27.784494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.16.40.254:38745)
0812 18:56:27.785835 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:27.785844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:27.785857 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:56:27.785893 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:27.785896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:27.785942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:27.786024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:27.787090 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.787698 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.788655 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:27.789002 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:27.789122 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:27.791105 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:27.791109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:27.791112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:27.791139 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:27.793113 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.795149 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.795152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.795154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.795251 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:27.795554 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:27.795558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:27.795560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:27.795883 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:27.796176 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:27.796327 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 18:56:27.796482 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":129,"threads_started":1}
I20260812 18:56:27.798352 31423 ts_manager.cc:194] Registered new tserver with Master: 1edf4fb1c248481fabd649732b73317f (127.16.40.195:44335)
I20260812 18:56:27.799980 31423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:40687
Aug 12 18:56:28 dist-test-slave-rnrw krb5kdc[30305](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786560988, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:28.264292 31645 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:28.264695 31645 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:28.265674 31419 ts_manager.cc:194] Registered new tserver with Master: 45b74a4c420e46a0b1692e35574c4c8e (127.16.40.193:33801)
I20260812 18:56:28.266584 31419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:44691
Aug 12 18:56:28 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560988, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:28.534178 31802 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:28.534516 31802 heartbeater.cc:511] Master 127.16.40.254:38745 requested a full tablet report, sending...
I20260812 18:56:28.537210 31419 ts_manager.cc:194] Registered new tserver with Master: f5d6d2fdce5f4d6aa737674ea919100f (127.16.40.194:35533)
I20260812 18:56:28.538041 31419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:55041
I20260812 18:56:28.547600 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:56:28.575678 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.557104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.16.40.254:38745)
0812 18:56:28.557902 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.557907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.565215 (+ 7308us) server_negotiation.cc:407] Connection header received
0812 18:56:28.565284 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.565288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.565354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.565446 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:28.566545 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.567420 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.568709 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.569077 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.569199 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.571160 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:28.571164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:28.571167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:28.571200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:28.573205 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:28.573750 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:28.573754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:28.573756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:28.573852 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:28.574240 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:28.574244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:28.574247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:28.574485 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:28.574620 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.575350 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.575494 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":100,"threads_started":1}
I20260812 18:56:28.580716 31419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:56:28.586164 31419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:56:28.607158 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.599427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:33801 (local address 127.0.0.1:51874)
0812 18:56:28.600778 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.600833 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:28.600970 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:28.601960 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:28.601967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:28.601993 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:28.602159 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.602166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.603801 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.603805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.605258 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.605269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.606607 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.606611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.606739 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.606787 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 18:56:28.606875 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 18:56:28.606995 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":137,"threads_started":1}
I20260812 18:56:28.608053 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.599540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.16.40.193:33801)
0812 18:56:28.601449 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.601455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.601487 (+ 32us) server_negotiation.cc:407] Connection header received
0812 18:56:28.601563 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.601568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.601776 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.602321 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:28.602456 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.603664 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.605433 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.606371 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.607592 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.607756 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.607808 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.607903 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1812,"thread_start_us":92,"threads_started":1}
I20260812 18:56:28.609508 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.600627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35533 (local address 127.0.0.1:38172)
0812 18:56:28.601659 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.601672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:28.601765 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:28.602443 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:28.602447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:28.602466 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:28.602560 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.602566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.604857 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.604861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.606045 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.606057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.609116 (+ 3059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.609120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.609211 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.609235 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:56:28.609306 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 18:56:28.609372 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260812 18:56:28.611320 31571 tablet_service.cc:1467] Processing CreateTablet for tablet 8b176774c8894393802fdb7e2e604ed6 (DEFAULT_TABLE table=test-table [id=b9f4510a008a4b2a85e17afbc85a7d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:28.612259 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.600768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.16.40.194:35533)
0812 18:56:28.601263 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.601270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.601861 (+ 591us) server_negotiation.cc:407] Connection header received
0812 18:56:28.601957 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.601963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.602150 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.602296 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:28.602697 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.604711 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.606195 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.608796 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.608902 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.609049 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.612007 (+ 2958us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.612126 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":89,"threads_started":1}
I20260812 18:56:28.612594 31571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b176774c8894393802fdb7e2e604ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:28.616706 31726 tablet_service.cc:1467] Processing CreateTablet for tablet 8b176774c8894393802fdb7e2e604ed6 (DEFAULT_TABLE table=test-table [id=b9f4510a008a4b2a85e17afbc85a7d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:28.617545 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.599466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:44335 (local address 127.0.0.1:54542)
0812 18:56:28.601385 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.601399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:28.601488 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:28.610330 (+ 8842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:28.610333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:28.610355 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:28.610440 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.610445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.613677 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.613681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.614811 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.614826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.616984 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.616988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.617108 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.617142 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 18:56:28.617195 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 18:56:28.617392 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1852,"thread_start_us":110,"threads_started":1}
I20260812 18:56:28.617936 31726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b176774c8894393802fdb7e2e604ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:28.618520 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.599634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.16.40.195:44335)
0812 18:56:28.607892 (+ 8258us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.607899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.607940 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:56:28.608023 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.608028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.610015 (+ 1987us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.610175 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:28.610734 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.613382 (+ 2648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.615001 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.616817 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.618080 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.618242 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.618298 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.618389 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8133,"thread_start_us":89,"threads_started":1}
I20260812 18:56:28.622149 31876 tablet_service.cc:1467] Processing CreateTablet for tablet 8b176774c8894393802fdb7e2e604ed6 (DEFAULT_TABLE table=test-table [id=b9f4510a008a4b2a85e17afbc85a7d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:28.623487 31876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b176774c8894393802fdb7e2e604ed6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:28.630218 31990 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Bootstrap starting.
I20260812 18:56:28.630728 31991 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Bootstrap starting.
I20260812 18:56:28.633502 31992 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Bootstrap starting.
I20260812 18:56:28.635030 31991 tablet_bootstrap.cc:654] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:28.635903 31992 tablet_bootstrap.cc:654] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:28.636191 31990 tablet_bootstrap.cc:654] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:28.636976 31992 log.cc:824] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:28.639271 31991 log.cc:824] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:28.639637 31992 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: No bootstrap required, opened a new log
I20260812 18:56:28.639631 31990 log.cc:824] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:28.639879 31992 ts_tablet_manager.cc:1397] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:56:28.643556 31990 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: No bootstrap required, opened a new log
I20260812 18:56:28.643965 31990 ts_tablet_manager.cc:1397] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260812 18:56:28.645161 31992 raft_consensus.cc:348] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.645510 31992 raft_consensus.cc:374] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:28.645569 31992 raft_consensus.cc:729] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edf4fb1c248481fabd649732b73317f, State: Initialized, Role: FOLLOWER
I20260812 18:56:28.646134 31992 consensus_queue.cc:260] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.647384 31992 ts_tablet_manager.cc:1428] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 18:56:28.647892 31952 heartbeater.cc:503] Master 127.16.40.254:38745 was elected leader, sending a full tablet report...
I20260812 18:56:28.650303 31990 raft_consensus.cc:348] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.650595 31990 raft_consensus.cc:374] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:28.650655 31990 raft_consensus.cc:729] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d6d2fdce5f4d6aa737674ea919100f, State: Initialized, Role: FOLLOWER
I20260812 18:56:28.651221 31990 consensus_queue.cc:260] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.652515 31990 ts_tablet_manager.cc:1428] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 18:56:28.653513 31802 heartbeater.cc:503] Master 127.16.40.254:38745 was elected leader, sending a full tablet report...
I20260812 18:56:28.656749 31991 tablet_bootstrap.cc:492] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: No bootstrap required, opened a new log
I20260812 18:56:28.657051 31991 ts_tablet_manager.cc:1397] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20260812 18:56:28.660745 31991 raft_consensus.cc:348] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.661504 31991 raft_consensus.cc:374] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:28.661576 31991 raft_consensus.cc:729] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b74a4c420e46a0b1692e35574c4c8e, State: Initialized, Role: FOLLOWER
I20260812 18:56:28.662184 31991 consensus_queue.cc:260] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.662972 31645 heartbeater.cc:503] Master 127.16.40.254:38745 was elected leader, sending a full tablet report...
I20260812 18:56:28.663451 31991 ts_tablet_manager.cc:1428] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:56:28.728945 31997 raft_consensus.cc:482] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:28.729249 31997 raft_consensus.cc:504] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
W20260812 18:56:28.730507 31647 tablet.cc:2369] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:56:28.730930 31997 leader_election.cc:288] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b74a4c420e46a0b1692e35574c4c8e (127.16.40.193:33801), 1edf4fb1c248481fabd649732b73317f (127.16.40.195:44335)
I20260812 18:56:28.737505 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.731757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:37905 (local address 127.16.40.193:33801)
0812 18:56:28.731880 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.731885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.732676 (+ 791us) server_negotiation.cc:407] Connection header received
0812 18:56:28.732864 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.732867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.732911 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.732955 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:28.733534 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.734255 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.736040 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.736710 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.736811 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.736866 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.737282 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.737367 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 18:56:28.737816 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.731762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:42443 (local address 127.16.40.195:44335)
0812 18:56:28.731902 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.731908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.732350 (+ 442us) server_negotiation.cc:407] Connection header received
0812 18:56:28.732621 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.732625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.732686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.732740 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:28.733477 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.734293 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.736334 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.737035 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.737139 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.737196 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.737612 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.737691 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:56:28.738360 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.731437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:33801 (local address 127.16.40.194:37905)
0812 18:56:28.732526 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.732564 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:28.732754 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:28.733079 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:28.733082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:28.733111 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:28.733394 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.733401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.734400 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.734403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.735888 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.735899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.736969 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.736972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.737134 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.737159 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:56:28.738054 (+ 895us) client_negotiation.cc:240] Negotiation successful
0812 18:56:28.738143 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":34,"thread_start_us":755,"threads_started":1}
I20260812 18:56:28.738401 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.731395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:44335 (local address 127.16.40.194:42443)
0812 18:56:28.732231 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.732250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:28.732446 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:28.733010 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:28.733014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:28.733035 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:28.733320 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.733330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.734500 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.734504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.736177 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:28.736188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.737303 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.737307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:28.737461 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.737485 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:56:28.738022 (+ 537us) client_negotiation.cc:240] Negotiation successful
0812 18:56:28.738119 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":41,"spinlock_wait_cycles":21248,"thread_start_us":618,"threads_started":1}
I20260812 18:56:28.739424 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b176774c8894393802fdb7e2e604ed6" candidate_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b74a4c420e46a0b1692e35574c4c8e" is_pre_election: true
I20260812 18:56:28.739506 31889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b176774c8894393802fdb7e2e604ed6" candidate_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edf4fb1c248481fabd649732b73317f" is_pre_election: true
I20260812 18:56:28.739816 31594 raft_consensus.cc:2454] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d6d2fdce5f4d6aa737674ea919100f in term 0.
I20260812 18:56:28.739853 31889 raft_consensus.cc:2454] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5d6d2fdce5f4d6aa737674ea919100f in term 0.
I20260812 18:56:28.740413 31676 leader_election.cc:302] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b74a4c420e46a0b1692e35574c4c8e, f5d6d2fdce5f4d6aa737674ea919100f; no voters:
I20260812 18:56:28.740774 31997 raft_consensus.cc:2781] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:56:28.740893 31997 raft_consensus.cc:482] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:56:28.740962 31997 raft_consensus.cc:3037] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:28.743108 31997 raft_consensus.cc:504] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.743770 31997 leader_election.cc:288] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [CANDIDATE]: Term 1 election: Requested vote from peers 45b74a4c420e46a0b1692e35574c4c8e (127.16.40.193:33801), 1edf4fb1c248481fabd649732b73317f (127.16.40.195:44335)
I20260812 18:56:28.744472 31889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b176774c8894393802fdb7e2e604ed6" candidate_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edf4fb1c248481fabd649732b73317f"
I20260812 18:56:28.744509 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b176774c8894393802fdb7e2e604ed6" candidate_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b74a4c420e46a0b1692e35574c4c8e"
I20260812 18:56:28.744668 31889 raft_consensus.cc:3037] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:28.744736 31594 raft_consensus.cc:3037] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:28.746618 31889 raft_consensus.cc:2454] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d6d2fdce5f4d6aa737674ea919100f in term 1.
I20260812 18:56:28.746793 31594 raft_consensus.cc:2454] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d6d2fdce5f4d6aa737674ea919100f in term 1.
I20260812 18:56:28.747087 31675 leader_election.cc:302] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1edf4fb1c248481fabd649732b73317f, f5d6d2fdce5f4d6aa737674ea919100f; no voters:
I20260812 18:56:28.747422 31997 raft_consensus.cc:2781] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:28.747668 31997 raft_consensus.cc:686] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 1 LEADER]: Becoming Leader. State: Replica: f5d6d2fdce5f4d6aa737674ea919100f, State: Running, Role: LEADER
I20260812 18:56:28.748222 31997 consensus_queue.cc:237] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } }
I20260812 18:56:28.752425 31419 catalog_manager.cc:5495] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5d6d2fdce5f4d6aa737674ea919100f (127.16.40.194). New cstate: current_term: 1 leader_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d6d2fdce5f4d6aa737674ea919100f" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35533 } health_report { overall_health: HEALTHY } } }
W20260812 18:56:28.757283 31803 tablet.cc:2369] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:56:28.777866 31954 tablet.cc:2369] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:56:28.803879 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.798197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.16.40.194:35533)
0812 18:56:28.798344 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:28.798352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:28.798370 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:56:28.798478 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:28.798482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:28.798555 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:28.798684 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:56:28.799120 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.799888 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.801290 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:28.801653 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:28.801761 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:28.802267 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:56:28.802915 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:56:28.803063 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:28.803604 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 18:56:28.803683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 18:56:28.818187 31594 raft_consensus.cc:1261] T 8b176774c8894393802fdb7e2e604ed6 P 45b74a4c420e46a0b1692e35574c4c8e [term 1 FOLLOWER]: Refusing update from remote peer f5d6d2fdce5f4d6aa737674ea919100f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:28.819531 31889 raft_consensus.cc:1261] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 1 FOLLOWER]: Refusing update from remote peer f5d6d2fdce5f4d6aa737674ea919100f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:28.820627 32002 consensus_queue.cc:1035] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b74a4c420e46a0b1692e35574c4c8e" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 18:56:28.821148 32007 consensus_queue.cc:1035] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edf4fb1c248481fabd649732b73317f" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:28.840611 32008 mvcc.cc:205] Tried to move back new op lower bound from 7317753810188075008 to 7317753809916727296. Current Snapshot: MvccSnapshot[applied={T|T < 7317753810188075008}]
I20260812 18:56:28.843609 32009 mvcc.cc:205] Tried to move back new op lower bound from 7317753810188075008 to 7317753809916727296. Current Snapshot: MvccSnapshot[applied={T|T < 7317753810188075008}]
I20260812 18:56:28.852895 31419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:28.853121 31419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36516:
table { table_name: "test-table" } modify_external_catalogs: true
Aug 12 18:56:28 dist-test-slave-rnrw krb5kdc[30305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560988, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for HTTP/127.16.40.212@KRBTEST.COM
I20260812 18:56:28.857208 31419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5426513e0b9f47a2bc6207220a4d4184: Sending DeleteTablet for 3 replicas of tablet 8b176774c8894393802fdb7e2e604ed6
I20260812 18:56:28.859069 31571 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b176774c8894393802fdb7e2e604ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:28 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:51874
I20260812 18:56:28.859786 31876 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b176774c8894393802fdb7e2e604ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:28 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:54542
I20260812 18:56:28.861290 31726 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b176774c8894393802fdb7e2e604ed6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:28 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:38172
I20260812 18:56:28.862947 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31465
I20260812 18:56:28.872020 32018 tablet_replica.cc:330] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: stopping tablet replica
I20260812 18:56:28.878557 32019 tablet_replica.cc:330] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: stopping tablet replica
I20260812 18:56:28.879038 32019 raft_consensus.cc:2229] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:56:28.879853 32019 raft_consensus.cc:2258] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:28.879350 32018 raft_consensus.cc:2229] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:28.880481 32018 raft_consensus.cc:2258] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:28.882921 32019 ts_tablet_manager.cc:1905] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:28.886102 32019 ts_tablet_manager.cc:1918] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:28.886229 32019 log.cc:1196] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/8b176774c8894393802fdb7e2e604ed6
I20260812 18:56:28.886579 32019 ts_tablet_manager.cc:1939] T 8b176774c8894393802fdb7e2e604ed6 P f5d6d2fdce5f4d6aa737674ea919100f: Deleting consensus metadata
I20260812 18:56:28.888105 31403 catalog_manager.cc:4841] TS f5d6d2fdce5f4d6aa737674ea919100f (127.16.40.194:35533): tablet 8b176774c8894393802fdb7e2e604ed6 (table test-table [id=b9f4510a008a4b2a85e17afbc85a7d04]) successfully deleted
I20260812 18:56:28.889829 32018 ts_tablet_manager.cc:1905] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:28.892935 32018 ts_tablet_manager.cc:1918] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:28.893074 32018 log.cc:1196] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/8b176774c8894393802fdb7e2e604ed6
I20260812 18:56:28.893448 32018 ts_tablet_manager.cc:1939] T 8b176774c8894393802fdb7e2e604ed6 P 1edf4fb1c248481fabd649732b73317f: Deleting consensus metadata
I20260812 18:56:28.894945 31405 catalog_manager.cc:4841] TS 1edf4fb1c248481fabd649732b73317f (127.16.40.195:44335): tablet 8b176774c8894393802fdb7e2e604ed6 (table test-table [id=b9f4510a008a4b2a85e17afbc85a7d04]) successfully deleted
W20260812 18:56:28.898432 31406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:33801 (error 108)
I20260812 18:56:28.899533 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31650
I20260812 18:56:28.905364 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:28.904883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:33801 (local address 127.0.0.1:51878)
0812 18:56:28.905147 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:28.905249 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260812 18:56:28.905609 31406 catalog_manager.cc:4584] TS 45b74a4c420e46a0b1692e35574c4c8e (127.16.40.193:33801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b176774c8894393802fdb7e2e604ed6: Network error: Client connection negotiation failed: client connection to 127.16.40.193:33801: connect: Connection refused (error 111)
I20260812 18:56:28.914675 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31806
I20260812 18:56:28.928275 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 31383
I20260812 18:56:28.945235 16547 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 18:56:29.291278 16547 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 18:56:29.291416 16547 mini_ranger.cc:66] Stopping Ranger...
I20260812 18:56:29.637486 16547 mini_ranger.cc:68] Stopped Ranger
2026-08-12 18:56:29.637 UTC [30339] LOG: received smart shutdown request
2026-08-12 18:56:29.642 UTC [30339] LOG: background worker "logical replication launcher" (PID 30348) exited with exit code 1
2026-08-12 18:56:29.642 UTC [30343] LOG: shutting down
2026-08-12 18:56:29.833 UTC [30339] LOG: database system is shut down
2026-08-12T18:56:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55951 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:56:30 dist-test-slave-rnrw krb5kdc[32032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:56:30 dist-test-slave-rnrw krb5kdc[32032](info): set up 2 sockets
Aug 12 18:56:30 dist-test-slave-rnrw krb5kdc[32032](info): commencing operation
krb5kdc: starting...
W20260812 18:56:31.126165 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:56:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:56:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:56:31.272310 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38673
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:31.391594 32047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:31.391918 32047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:31.391970 32047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:31.396162 32047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:31.396248 32047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:31.396270 32047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:31.396288 32047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:31.396304 32047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:31.401983 32047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:31.403148 32047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:31.404247 32047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:31.414191 32052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.414191 32056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.414217 32053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:31.415333 32047 server_base.cc:1034] running on GCE node
I20260812 18:56:31.416093 32047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:31.417503 32047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:31.418741 32047 hybrid_clock.cc:648] HybridClock initialized: now 1786560991418723 us; error 33 us; skew 500 ppm
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:31.422020 32047 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:31.423118 32047 webserver.cc:460] Webserver started at http://127.16.40.254:42881/ using document root <none> and password file <none>
I20260812 18:56:31.423696 32047 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:31.423760 32047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:31.423943 32047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:31.426162 32047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "6cd68696311a43aa983c86475c9c5388"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "48b9bf8415485f010e50763ae0d8777f"
server_key_iv: "5ce5fbabf6bb67d4c9bdf316e4074246"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.426723 32047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "6cd68696311a43aa983c86475c9c5388"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "48b9bf8415485f010e50763ae0d8777f"
server_key_iv: "5ce5fbabf6bb67d4c9bdf316e4074246"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.430971 32047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:56:31.433912 32065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:31.435555 32047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:31.435698 32047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "6cd68696311a43aa983c86475c9c5388"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "48b9bf8415485f010e50763ae0d8777f"
server_key_iv: "5ce5fbabf6bb67d4c9bdf316e4074246"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.435810 32047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:31.451949 32047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:31.455935 32047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:31.456126 32047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:31.464417 32047 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38673
I20260812 18:56:31.464438 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38673 every 8 connection(s)
I20260812 18:56:31.465533 32047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:31.468784 32131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:31.471050 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32047
I20260812 18:56:31.471201 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:56:31.471506 16547 external_mini_cluster.cc:1428] Setting key 629395ae3f62752b247a5c10caf25d55
I20260812 18:56:31.474247 32131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.253@KRBTEST.COM" created.
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:31.493804 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.478367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.16.40.254:38673)
0812 18:56:31.478664 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.478673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.478712 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:56:31.479967 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.479971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.480178 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.480340 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.481650 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.482643 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.483818 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.484373 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.484528 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.487696 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.487710 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.487733 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.487775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.489862 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.490640 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.490646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.490647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.490833 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.491291 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.491294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.491295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.491795 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.491916 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.492491 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.492872 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260812 18:56:31.493808 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.477561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38673 (local address 127.0.0.1:51466)
0812 18:56:31.478210 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.478285 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.479590 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.480424 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.480431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.480958 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.481406 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.481433 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.482849 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.482852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.483665 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.483681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.484702 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.484709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.484842 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.485611 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.485632 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.487484 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.490015 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.490031 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.490047 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.490440 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.490978 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.490984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.490988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.491178 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.491939 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.491951 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.492177 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.492421 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.492985 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":89,"thread_start_us":121,"threads_started":1}
I20260812 18:56:31.496803 32131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.497751 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.497358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44484)
0812 18:56:31.497501 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.497674 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 18:56:31.498076 32066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
I20260812 18:56:31.499115 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.498844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44498)
0812 18:56:31.498998 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.499044 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 18:56:31.499320 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:34405: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
I20260812 18:56:31.507692 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:34405
--webserver_interface=127.16.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:31.548662 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } attempt: 1
I20260812 18:56:31.549700 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.549331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44502)
0812 18:56:31.549509 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.549615 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 18:56:31.550490 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.550264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44506)
0812 18:56:31.550379 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.550433 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 18:56:31.550634 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:34405: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
I20260812 18:56:31.618840 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } attempt: 2
I20260812 18:56:31.619773 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.619457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44522)
0812 18:56:31.619614 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.619686 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 18:56:31.620522 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.620295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44538)
0812 18:56:31.620409 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.620465 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 18:56:31.620697 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:34405: Network error: Client connection negotiation failed: client connection to 127.16.40.253:34405: connect: Connection refused (error 111)
W20260812 18:56:31.630324 32138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:31.630769 32138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:31.630895 32138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:31.635418 32138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:31.635535 32138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:31.635584 32138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:31.635625 32138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:31.635663 32138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:31.641841 32138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:34405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:31.643177 32138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:31.644310 32138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:31.653669 32145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.653658 32146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.653658 32149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:31.653882 32138 server_base.cc:1034] running on GCE node
I20260812 18:56:31.654615 32138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:31.655943 32138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:31.657152 32138 hybrid_clock.cc:648] HybridClock initialized: now 1786560991657136 us; error 30 us; skew 500 ppm
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:31.660321 32138 init.cc:377] Logged in from keytab as kudu/127.16.40.253@KRBTEST.COM (short username kudu)
I20260812 18:56:31.661386 32138 webserver.cc:460] Webserver started at http://127.16.40.253:34885/ using document root <none> and password file <none>
I20260812 18:56:31.661970 32138 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:31.662038 32138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:31.662230 32138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:31.664368 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data/instance:
uuid: "50fb4fd4877a49709106319c861e0403"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "33cf883e2703bd12bc9cb9eadf61ca05"
server_key_iv: "71997a39f69f8a2c63bdeeb30121b6ab"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.665015 32138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal/instance:
uuid: "50fb4fd4877a49709106319c861e0403"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "33cf883e2703bd12bc9cb9eadf61ca05"
server_key_iv: "71997a39f69f8a2c63bdeeb30121b6ab"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.669227 32138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:56:31.671782 32157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:31.673182 32138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:31.673311 32138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "50fb4fd4877a49709106319c861e0403"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "33cf883e2703bd12bc9cb9eadf61ca05"
server_key_iv: "71997a39f69f8a2c63bdeeb30121b6ab"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.673409 32138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:31.694526 32138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:31.698326 32138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:31.698531 32138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:31.706745 32138 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.253:34405
I20260812 18:56:31.706753 32215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.253:34405 every 8 connection(s)
I20260812 18:56:31.707864 32138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
I20260812 18:56:31.711337 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:31.711905 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } attempt: 3
I20260812 18:56:31.716253 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32138
I20260812 18:56:31.716401 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-1/wal/instance
I20260812 18:56:31.716811 16547 external_mini_cluster.cc:1428] Setting key 19e5a2140d29973896b693c0f54be02f
I20260812 18:56:31.717092 32216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.252@KRBTEST.COM; defaulting to no policy
I20260812 18:56:31.728344 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.712608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44552)
0812 18:56:31.712788 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.712808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.712919 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.716042 (+ 3123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.716046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.716304 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.716483 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.716494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.718490 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.718497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.719491 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.719508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.720710 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.720717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.720874 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.721511 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.721531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.723156 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.726096 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.726100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.726103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.726335 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.726735 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.726738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.726740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.726847 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.727667 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.727673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.727848 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.727979 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.728203 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 18:56:31.730564 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.713342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.16.40.253:34405)
0812 18:56:31.713888 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.713909 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.713955 (+ 46us) server_negotiation.cc:407] Connection header received
0812 18:56:31.715101 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.715130 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.715625 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.716046 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.716793 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.718307 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.719707 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.720489 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.720762 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.723370 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.723376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.723386 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.723419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.725949 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.726478 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.726485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.726490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.726626 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.726972 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.726975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.726976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.727506 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.727679 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.728349 (+ 670us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.729121 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.252@KRBTEST.COM" created.
I20260812 18:56:31.732795 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.719560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38673 (local address 127.0.0.1:51468)
0812 18:56:31.720073 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.720141 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.720342 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.720867 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.720873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.721415 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.721609 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.721616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.722483 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.722488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.723753 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.723768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.724352 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.724358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.724526 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.725319 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.725338 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.727288 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.729625 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.729635 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.729641 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.729941 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.730271 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.730281 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.730282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.730391 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.732004 (+ 1613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.732019 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.732166 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.732335 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.732598 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":156,"threads_started":1}
I20260812 18:56:31.733381 32131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.733771 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.719569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.16.40.254:38673)
0812 18:56:31.719730 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.719736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.720226 (+ 490us) server_negotiation.cc:407] Connection header received
0812 18:56:31.720511 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.720515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.720615 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.720699 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.721749 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.722320 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.723914 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.724210 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.724304 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.727474 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.727478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.727481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.727505 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.729492 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.730063 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.730067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.730069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.730165 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.730495 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.730498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.730500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.730757 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.730863 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.733353 (+ 2490us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.733532 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:56:31.734318 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.733987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37340)
0812 18:56:31.734154 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.734231 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 18:56:31.735003 32216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.735494 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.734958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37346)
0812 18:56:31.735342 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.735404 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":110,"threads_started":1}
W20260812 18:56:31.735720 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:56:31.744298 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.735575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44558)
0812 18:56:31.735699 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.735714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.735803 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.736050 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.736053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.736242 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.736315 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.736320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.737154 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.737161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.737738 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.737747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.738234 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.738237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.738319 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.738872 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.738889 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.740792 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.742706 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.742711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.742713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.742921 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.743212 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.743217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.743219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.743316 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.743720 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.743723 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.743820 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.743929 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.744099 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 18:56:31.744297 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.735756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.16.40.253:34405)
0812 18:56:31.735860 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.735865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.735878 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:56:31.735906 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.735909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.735952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.736043 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.736431 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.737044 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.737863 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.738119 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.738233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.740921 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.740925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.740928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.740952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.742556 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.743032 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.743036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.743037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.743117 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.743408 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.743411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.743413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.743614 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.743721 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.743990 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.744162 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":36}
I20260812 18:56:31.745301 32216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.746181 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.745886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37348)
0812 18:56:31.746031 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.746115 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 18:56:31.746716 32158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
I20260812 18:56:31.747862 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.747590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37358)
0812 18:56:31.747725 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.747776 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 18:56:31.748180 32216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
I20260812 18:56:31.752272 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:43961
--webserver_interface=127.16.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:31.766912 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 1
I20260812 18:56:31.768117 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.767565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37372)
0812 18:56:31.767872 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.767959 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260812 18:56:31.769165 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.768846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37376)
0812 18:56:31.768997 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.769065 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 18:56:31.769346 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
I20260812 18:56:31.797529 32216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 1
I20260812 18:56:31.798609 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.798198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37380)
0812 18:56:31.798411 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.798499 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 18:56:31.799535 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.799263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37388)
0812 18:56:31.799389 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.799449 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 18:56:31.799721 32216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
I20260812 18:56:31.844601 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 2
I20260812 18:56:31.845634 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.845208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37392)
0812 18:56:31.845429 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.845520 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 18:56:31.846611 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.846313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37400)
0812 18:56:31.846461 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.846534 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 18:56:31.846822 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
I20260812 18:56:31.867915 32216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 2
I20260812 18:56:31.868902 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.868514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37402)
0812 18:56:31.868747 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.868818 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 18:56:31.869834 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.869583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37404)
0812 18:56:31.869712 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.869772 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 18:56:31.869962 32216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
W20260812 18:56:31.878896 32227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:31.879251 32227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:31.879354 32227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:31.883852 32227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:31.883949 32227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:31.883994 32227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:31.884032 32227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:31.884064 32227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:31.889863 32227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38673,127.16.40.253:34405,127.16.40.252:43961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:31.891182 32227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:31.892303 32227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:31.901716 32235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.901727 32233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:31.901716 32232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:31.902236 32227 server_base.cc:1034] running on GCE node
I20260812 18:56:31.902963 32227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:31.904270 32227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:31.905556 32227 hybrid_clock.cc:648] HybridClock initialized: now 1786560991905483 us; error 46 us; skew 500 ppm
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:31.908937 32227 init.cc:377] Logged in from keytab as kudu/127.16.40.252@KRBTEST.COM (short username kudu)
I20260812 18:56:31.910169 32227 webserver.cc:460] Webserver started at http://127.16.40.252:34843/ using document root <none> and password file <none>
I20260812 18:56:31.910844 32227 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:31.910951 32227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:31.911219 32227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:31.913669 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data/instance:
uuid: "737d23918bc84e849e1ff1e75bd81097"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "19015db68e8ee99051da29a19969c35c"
server_key_iv: "233c859fd0fb928d092ddd0687255462"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.914336 32227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal/instance:
uuid: "737d23918bc84e849e1ff1e75bd81097"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "19015db68e8ee99051da29a19969c35c"
server_key_iv: "233c859fd0fb928d092ddd0687255462"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.918692 32227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 18:56:31.921625 32243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:31.923022 32227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:31.923202 32227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "737d23918bc84e849e1ff1e75bd81097"
format_stamp: "Formatted at 2026-08-12 18:56:31 on dist-test-slave-rnrw"
server_key: "19015db68e8ee99051da29a19969c35c"
server_key_iv: "233c859fd0fb928d092ddd0687255462"
server_key_version: "encryptionkey@0"
I20260812 18:56:31.923341 32227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:31.935420 32227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:31.939217 32227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:31.939433 32227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:31.946051 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 3
I20260812 18:56:31.946951 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.946651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37416)
0812 18:56:31.946795 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.946871 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 18:56:31.947897 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.947562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37428)
0812 18:56:31.947750 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.947809 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 18:56:31.947981 32227 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.252:43961
I20260812 18:56:31.947984 32310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.252:43961 every 8 connection(s)
W20260812 18:56:31.948089 32131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43961: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43961: connect: Connection refused (error 111)
I20260812 18:56:31.949214 32227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
I20260812 18:56:31.950820 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32227
I20260812 18:56:31.950973 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786560811542710-16547-0/minicluster-data/master-2/wal/instance
I20260812 18:56:31.951267 16547 external_mini_cluster.cc:1428] Setting key 332b779ca4a4c3ba7bf0038bb343e976
I20260812 18:56:31.952613 32312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:31.956698 32312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:56:31.961161 32216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 3
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:56:31.967389 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.952651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.16.40.252:43961)
0812 18:56:31.953110 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.953149 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.953186 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:56:31.953820 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.953835 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.954177 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.954524 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.955416 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.956434 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.957437 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.957931 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.958115 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.961023 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.961029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.961040 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.961070 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.963467 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.964065 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.964071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.964076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.964204 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.964628 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.964632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.964633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.965272 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.965579 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.965901 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.966255 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":153,"threads_started":1}
I20260812 18:56:31.971235 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.958704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.16.40.254:38673)
0812 18:56:31.958850 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.958853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.959232 (+ 379us) server_negotiation.cc:407] Connection header received
0812 18:56:31.959413 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.959416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.959469 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.959556 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.960294 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.960934 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.962459 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.962835 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.962958 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.966216 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.966221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.966224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.966258 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.968631 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.969428 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.969432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.969434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.969536 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.969905 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.969909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.969910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.970213 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.970308 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.970879 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.971024 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:56:31.971927 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.958636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38673 (local address 127.0.0.1:51478)
0812 18:56:31.959106 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.959149 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.959301 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.959612 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.959626 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.960019 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.960160 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.960166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.961067 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.961071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.962068 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.962078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.962972 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.962976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.963094 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.964142 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.964158 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.966012 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.968798 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.968810 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.968818 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.969261 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.969663 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.969670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.969672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.969777 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.970406 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.970412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.970588 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.971427 (+ 839us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.971683 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":189,"threads_started":1}
I20260812 18:56:31.972205 32223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.961652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37454)
0812 18:56:31.961818 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.961834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.961914 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.962282 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.962285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.962464 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.962549 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.962555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.963471 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.963475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.964307 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.964322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.964948 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.964951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.965040 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.966101 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.966124 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.967508 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.970403 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.970407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.970409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.970701 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.971047 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.971051 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.971053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.971187 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.971629 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.971634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.971753 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.971873 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.972040 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 18:56:31.972223 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.961685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.16.40.252:43961)
0812 18:56:31.962036 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.962044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.962059 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:56:31.962098 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.962102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.962159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.962261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.962669 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.963345 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.964446 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.964815 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.964931 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.967685 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.967688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.967690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.967737 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.970274 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.970834 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.970838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.970839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.970952 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.971295 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.971301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.971304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.971535 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.971663 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.971937 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.972054 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260812 18:56:31.973611 32312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.976208 32216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: Bootstrap starting.
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:31.979418 32216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: Neither blocks nor log segments found. Creating new log.
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:56:31.980531 32216 log.cc:824] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:31.982697 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.970834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.16.40.254:38673)
0812 18:56:31.971329 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.971346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.971360 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:56:31.971398 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.971401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.971448 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.972387 (+ 939us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.972514 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.973338 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.974334 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.974728 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.975258 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.977950 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.977954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.977957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.977986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.980107 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.980679 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.980683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.980685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.980794 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.981584 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.981588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.981589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.981806 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.982038 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.982214 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.982346 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":90,"threads_started":1}
I20260812 18:56:31.983152 32216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: No bootstrap required, opened a new log
I20260812 18:56:31.984090 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.974148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:34405 (local address 127.0.0.1:44564)
0812 18:56:31.974516 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.974530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.974628 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.974990 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.974993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.975550 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.975692 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.975702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.976855 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.976858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.977732 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.977745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.978353 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.978357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.978454 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.978894 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.978923 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.980300 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.982345 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.982349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.982351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.982591 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.982941 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.982944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.982946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.983043 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.983529 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.983534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.983650 (+ 116us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.983774 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.983932 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296}
I20260812 18:56:31.984136 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.974390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.16.40.253:34405)
0812 18:56:31.974540 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.974544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.974579 (+ 35us) server_negotiation.cc:407] Connection header received
0812 18:56:31.974721 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.974724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.974780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.974859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.975849 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.976711 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.977882 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.978114 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.978172 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.980455 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.980459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.980462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.980488 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.982204 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.982711 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.982717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.982720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.982818 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.983178 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.983182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.983184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.983412 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.983512 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.983846 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.983974 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 18:56:31.985177 32312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:56:31.987855 32216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:31.988129 32216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:31.988219 32216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fb4fd4877a49709106319c861e0403, State: Initialized, Role: FOLLOWER
I20260812 18:56:31.988791 32216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:31.989419 32322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:31.989989 32322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:56:31.990521 32216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 18:56:31 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:56:31.995016 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.985802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37456)
0812 18:56:31.985986 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:31.985999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:31.986084 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:31.986400 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:31.986403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:31.986573 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:31.986658 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.986664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.987662 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.987665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.988399 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:31.988412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.989248 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.989251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:31.989326 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.989800 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:31.989814 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:31.991191 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:31.993331 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.993334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.993338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.993621 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.993919 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:31.993921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:31.993923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:31.994012 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:31.994468 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:31.994471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:31.994554 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 18:56:31.994651 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 18:56:31.994801 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 18:56:31.995498 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.985845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.16.40.252:43961)
0812 18:56:31.985958 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.985962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.986068 (+ 106us) server_negotiation.cc:407] Connection header received
0812 18:56:31.986179 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.986182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.986230 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.986307 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.986774 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.987538 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.988612 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.988998 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.989092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.991374 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.991382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.991387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.991420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.993202 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.993729 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.993732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.993733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.993813 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.994108 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.994111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.994113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.994337 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.994435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.995236 (+ 801us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.995357 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 18:56:31.995793 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:31.984180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.16.40.253:34405)
0812 18:56:31.984722 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:31.984726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:31.985036 (+ 310us) server_negotiation.cc:407] Connection header received
0812 18:56:31.985214 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:31.985217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:31.985262 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:31.985337 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:31.986260 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.986761 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.987556 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:31.987784 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:31.987894 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:31.991456 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:31.991462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:31.991465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:31.991495 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:31.993292 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.993877 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.993881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.993883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.993973 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:31.994298 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:31.994301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:31.994303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:31.994513 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:31.994707 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:31.995484 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 18:56:31.995634 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":140,"threads_started":1}
W20260812 18:56:31.997565 32335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:56:31.997696 32335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:56:31.995149 32138 master_runner.cc:186] Error getting master registration for 127.16.40.254:38673: OK, instance_id { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" instance_seqno: 1786560991462061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:56:31.999392 32138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 18:56:31.999698 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:56:32.002705 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: Bootstrap starting.
I20260812 18:56:32.005681 32312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:32.006809 32312 log.cc:824] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:32.014804 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:32.002286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.16.40.253:34405)
0812 18:56:32.002413 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:32.002418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:32.004897 (+ 2479us) server_negotiation.cc:407] Connection header received
0812 18:56:32.005102 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:32.005105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:32.005164 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:32.005219 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:32.006245 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.007025 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.008329 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.008763 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.008868 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:32.010762 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:32.010767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:32.010770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:32.010802 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:32.012799 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.013306 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.013310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.013312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.013405 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.013745 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.013760 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.013761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.013985 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:32.014084 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:32.014447 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 18:56:32.014579 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:56:32.014914 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:32.004720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.16.40.254:38673)
0812 18:56:32.004852 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:32.004857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:32.004872 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:56:32.004933 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:32.004935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:32.004985 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:32.005073 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:32.006163 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.006986 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.008006 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.008377 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.008504 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:32.010621 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:32.010625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:32.010628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:32.010660 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:32.012601 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.013194 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.013198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.013200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.013298 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.013622 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.013626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.013627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.013865 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:32.014012 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:32.014628 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 18:56:32.014761 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:56:32.014951 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: No bootstrap required, opened a new log
I20260812 18:56:32.019044 32312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.019308 32312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:32.019346 32312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737d23918bc84e849e1ff1e75bd81097, State: Initialized, Role: FOLLOWER
I20260812 18:56:32.019975 32312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.020641 32347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.020862 32347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.021719 32312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:32.024479 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:32.003024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.16.40.252:43961)
0812 18:56:32.003137 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:32.003142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:32.008211 (+ 5069us) server_negotiation.cc:407] Connection header received
0812 18:56:32.008390 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:32.008393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:32.008456 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:32.008510 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:32.009576 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.010397 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.011555 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.011940 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.012032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:32.020437 (+ 8405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:32.020442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:32.020445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:32.020479 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:32.022366 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.022980 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.022984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.022986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.023077 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.023449 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.023453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.023455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.023703 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:32.023867 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:32.024178 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 18:56:32.024313 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 18:56:32.029078 32361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:56:32.029259 32361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:56:32.026731 32227 master_runner.cc:186] Error getting master registration for 127.16.40.254:38673: OK, instance_id { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" instance_seqno: 1786560991462061 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:56:32.030135 32227 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 18:56:32.112259 32131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } attempt: 4
I20260812 18:56:32.113420 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:32.113615 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.114681 32347 leader_election.cc:288] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cd68696311a43aa983c86475c9c5388 (127.16.40.254:38673), 50fb4fd4877a49709106319c861e0403 (127.16.40.253:34405)
I20260812 18:56:32.116084 32188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737d23918bc84e849e1ff1e75bd81097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fb4fd4877a49709106319c861e0403" is_pre_election: true
I20260812 18:56:32.116458 32188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737d23918bc84e849e1ff1e75bd81097 in term 0.
I20260812 18:56:32.117059 32244 leader_election.cc:302] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fb4fd4877a49709106319c861e0403, 737d23918bc84e849e1ff1e75bd81097; no voters:
I20260812 18:56:32.117435 32347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:56:32.117522 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:56:32.117563 32347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:32.118818 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.119202 32347 leader_election.cc:288] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [CANDIDATE]: Term 1 election: Requested vote from peers 6cd68696311a43aa983c86475c9c5388 (127.16.40.254:38673), 50fb4fd4877a49709106319c861e0403 (127.16.40.253:34405)
Aug 12 18:56:32 dist-test-slave-rnrw krb5kdc[32032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560991, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:56:32.120119 32188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737d23918bc84e849e1ff1e75bd81097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fb4fd4877a49709106319c861e0403"
I20260812 18:56:32.120263 32188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:32.121759 32188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737d23918bc84e849e1ff1e75bd81097 in term 1.
I20260812 18:56:32.122263 32244 leader_election.cc:302] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fb4fd4877a49709106319c861e0403, 737d23918bc84e849e1ff1e75bd81097; no voters:
I20260812 18:56:32.122553 32347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:32.122831 32347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [term 1 LEADER]: Becoming Leader. State: Replica: 737d23918bc84e849e1ff1e75bd81097, State: Running, Role: LEADER
I20260812 18:56:32.123659 32347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.126382 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:32.112984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.16.40.252:43961)
0812 18:56:32.113131 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:32.113135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:32.113168 (+ 33us) server_negotiation.cc:407] Connection header received
0812 18:56:32.113332 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:32.113336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:32.113390 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:32.113467 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:32.114329 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.115126 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.116453 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.116832 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.116939 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:32.120559 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:32.120565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:32.120569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:32.120610 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:32.122697 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.123766 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.123771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.123773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.123952 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:32.124598 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:32.124602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:32.124604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:32.125059 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:32.125197 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:32.125828 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 18:56:32.126085 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:56:32.127245 32225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:32.112915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43961 (local address 127.0.0.1:37468)
0812 18:56:32.113059 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:32.113074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:32.113182 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:32.113655 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:32.113660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:32.114009 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:32.114162 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:32.114173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.115287 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.115293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:32.116183 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:32.116197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:32.117189 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:32.117193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:32.117300 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:32.118012 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:32.118029 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:32.120331 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:32.123074 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:32.123079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:32.123081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:32.123578 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:32.124235 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:32.124240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:32.124242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:32.124409 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:32.125323 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:32.125326 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:32.125508 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 18:56:32.126653 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0812 18:56:32.126975 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 18:56:32.127949 32347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737d23918bc84e849e1ff1e75bd81097. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.128249 32347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:32.131119 32131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: Bootstrap starting.
I20260812 18:56:32.135202 32365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:32.135619 32131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:32.137177 32365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:32.137359 32131 log.cc:824] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:32.140504 32131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: No bootstrap required, opened a new log
I20260812 18:56:32.146523 32131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.147018 32131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:32.147200 32131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cd68696311a43aa983c86475c9c5388, State: Initialized, Role: FOLLOWER
I20260812 18:56:32.147378 32188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [term 1 FOLLOWER]: Refusing update from remote peer 737d23918bc84e849e1ff1e75bd81097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:32.148340 32363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:32.148206 32131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } }
I20260812 18:56:32.153566 32093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:32.154347 32094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737d23918bc84e849e1ff1e75bd81097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd68696311a43aa983c86475c9c5388"
I20260812 18:56:32.154815 32322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737d23918bc84e849e1ff1e75bd81097. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.154959 32322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.155145 32093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 1 FOLLOWER]: Refusing update from remote peer 737d23918bc84e849e1ff1e75bd81097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:32.155426 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737d23918bc84e849e1ff1e75bd81097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cd68696311a43aa983c86475c9c5388" is_pre_election: true
I20260812 18:56:32.155642 32095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737d23918bc84e849e1ff1e75bd81097 in term 1.
I20260812 18:56:32.155936 32363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:32.160666 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.160856 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.160990 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737d23918bc84e849e1ff1e75bd81097. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.161085 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.153299 32131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:32.162671 32347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.162811 32347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:32.162935 32367 mvcc.cc:205] Tried to move back new op lower bound from 7317753823823581184 to 7317753823745671168. Current Snapshot: MvccSnapshot[applied={T|T < 7317753823823581184}]
I20260812 18:56:32.163436 32322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.163571 32322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.164076 32369 mvcc.cc:205] Tried to move back new op lower bound from 7317753823823581184 to 7317753823745671168. Current Snapshot: MvccSnapshot[applied={T|T < 7317753823823581184}]
I20260812 18:56:32.168170 32365 catalog_manager.cc:1347] Generated new cluster ID: c78e09d8bc4e4d0a8e77dc0bb72deb22
I20260812 18:56:32.168256 32365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:32.168483 32363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.168689 32363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:32.170915 32371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737d23918bc84e849e1ff1e75bd81097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cd68696311a43aa983c86475c9c5388" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38673 } } peers { permanent_uuid: "50fb4fd4877a49709106319c861e0403" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 34405 } } peers { permanent_uuid: "737d23918bc84e849e1ff1e75bd81097" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43961 } } }
I20260812 18:56:32.171061 32371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:56:32.178082 32389 catalog_manager.cc:1259] Loaded cluster ID: c78e09d8bc4e4d0a8e77dc0bb72deb22
I20260812 18:56:32.178179 32389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: loading cluster ID for follower catalog manager: success
W20260812 18:56:32.178807 32389 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260812 18:56:32.178926 32389 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:56:32.181959 32372 mvcc.cc:205] Tried to move back new op lower bound from 7317753823823581184 to 7317753823745671168. Current Snapshot: MvccSnapshot[applied={T|T < 7317753823823581184 or (T in {7317753823823581184})}]
I20260812 18:56:32.209899 32365 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:56:32.211301 32365 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:32.228950 32365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737d23918bc84e849e1ff1e75bd81097: Generated new TSK 0
I20260812 18:56:32.229969 32365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:56:32.999243 32335 catalog_manager.cc:1259] Loaded cluster ID: c78e09d8bc4e4d0a8e77dc0bb72deb22
I20260812 18:56:32.999341 32335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: loading cluster ID for follower catalog manager: success
I20260812 18:56:33.001451 32335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: acquiring CA information for follower catalog manager: success
I20260812 18:56:33.002799 32335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50fb4fd4877a49709106319c861e0403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:56:33.181509 32389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: acquiring CA information for follower catalog manager: success
I20260812 18:56:33.182813 32389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6cd68696311a43aa983c86475c9c5388: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:56:33.306293 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32047
I20260812 18:56:33.316915 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32138
I20260812 18:56:33.326301 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32227
2026-08-12T18:56:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3366 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:56:33 dist-test-slave-rnrw krb5kdc[32401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:56:33 dist-test-slave-rnrw krb5kdc[32401](info): set up 2 sockets
Aug 12 18:56:33 dist-test-slave-rnrw krb5kdc[32401](info): commencing operation
krb5kdc: starting...
W20260812 18:56:34.464504 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:56:34 dist-test-slave-rnrw krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:56:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:56:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:56:34.609143 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33811
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:34.731016 32416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:34.731348 32416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:34.731390 32416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:34.735673 32416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:34.735760 32416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:34.735781 32416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:34.735798 32416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:34.735814 32416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:34.742009 32416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:34.743294 32416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:34.744472 32416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:34.754149 32426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:34.754170 32424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:34.754351 32416 server_base.cc:1034] running on GCE node
W20260812 18:56:34.754128 32423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:34.755223 32416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:34.756582 32416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:34.757793 32416 hybrid_clock.cc:648] HybridClock initialized: now 1786560994757755 us; error 55 us; skew 500 ppm
Aug 12 18:56:34 dist-test-slave-rnrw krb5kdc[32401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560994, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:34.761044 32416 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:34.762023 32416 webserver.cc:460] Webserver started at http://127.16.40.254:42503/ using document root <none> and password file <none>
I20260812 18:56:34.762598 32416 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:34.762660 32416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:34.762838 32416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:34.765208 32416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "c9b1305997634026b35fd018a8e0f341"
format_stamp: "Formatted at 2026-08-12 18:56:34 on dist-test-slave-rnrw"
server_key: "af8a50903a29dcbc590417e55c6fa644"
server_key_iv: "33b0b43bc45e7f6b50ec77a579f9a87e"
server_key_version: "encryptionkey@0"
I20260812 18:56:34.765772 32416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "c9b1305997634026b35fd018a8e0f341"
format_stamp: "Formatted at 2026-08-12 18:56:34 on dist-test-slave-rnrw"
server_key: "af8a50903a29dcbc590417e55c6fa644"
server_key_iv: "33b0b43bc45e7f6b50ec77a579f9a87e"
server_key_version: "encryptionkey@0"
I20260812 18:56:34.770068 32416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 18:56:34.773241 32433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:34.774736 32416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:34.774905 32416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "c9b1305997634026b35fd018a8e0f341"
format_stamp: "Formatted at 2026-08-12 18:56:34 on dist-test-slave-rnrw"
server_key: "af8a50903a29dcbc590417e55c6fa644"
server_key_iv: "33b0b43bc45e7f6b50ec77a579f9a87e"
server_key_version: "encryptionkey@0"
I20260812 18:56:34.775053 32416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:34.789696 32416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:34.793282 32416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:34.793511 32416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:34.801383 32416 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33811
I20260812 18:56:34.801417 32494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33811 every 8 connection(s)
I20260812 18:56:34.802426 32416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:34.805696 32495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:34.807565 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32416
I20260812 18:56:34.807708 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:56:34.807996 16547 external_mini_cluster.cc:1428] Setting key 85a07aba1003f696732e3dcf76458c6e
I20260812 18:56:34.812358 32495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341: Bootstrap starting.
I20260812 18:56:34.814811 32495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:34.815599 32495 log.cc:824] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:56:34 dist-test-slave-rnrw krb5kdc[32401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:34.817771 32495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341: No bootstrap required, opened a new log
I20260812 18:56:34.821712 32495 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } }
I20260812 18:56:34.822005 32495 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:34.822096 32495 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b1305997634026b35fd018a8e0f341, State: Initialized, Role: FOLLOWER
I20260812 18:56:34.822540 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:34.809516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.16.40.254:33811)
0812 18:56:34.810008 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:34.810024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:34.810061 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:56:34.810744 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:34.810761 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:34.811096 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:34.811450 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:34.812299 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:34.813371 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:34.814224 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:34.814705 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:34.814884 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:34.817552 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:34.817557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:34.817572 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:34.817600 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:34.819371 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:34.819850 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:34.819856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:34.819861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:34.819978 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:34.820282 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:34.820285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:34.820287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:34.820785 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:34.821001 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:34.821334 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 18:56:34.821645 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":143,"threads_started":1}
I20260812 18:56:34.822643 32495 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } }
I20260812 18:56:34.822841 32495 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:34.822930 32495 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:34.823033 32495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:34.824018 32495 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } }
I20260812 18:56:34.824384 32495 leader_election.cc:302] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b1305997634026b35fd018a8e0f341; no voters:
I20260812 18:56:34.824793 32495 leader_election.cc:288] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:56:34.824900 32503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:34.825150 32503 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [term 1 LEADER]: Becoming Leader. State: Replica: c9b1305997634026b35fd018a8e0f341, State: Running, Role: LEADER
I20260812 18:56:34.825470 32503 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } }
I20260812 18:56:34.825726 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:56:34.826231 32495 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:34.827308 32504 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b1305997634026b35fd018a8e0f341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } } }
I20260812 18:56:34.827478 32504 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:34.827978 32505 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b1305997634026b35fd018a8e0f341. Latest consensus state: current_term: 1 leader_uuid: "c9b1305997634026b35fd018a8e0f341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b1305997634026b35fd018a8e0f341" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33811 } } }
I20260812 18:56:34.828104 32505 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:34.827968 32512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:34.831205 32512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:34.837390 32512 catalog_manager.cc:1347] Generated new cluster ID: 275b56bd30ec48f9aee26c688c9b02b4
I20260812 18:56:34.837493 32512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:34.869840 32512 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:56:34.870950 32512 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:34.881031 32512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b1305997634026b35fd018a8e0f341: Generated new TSK 0
I20260812 18:56:34.881958 32512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:56:34.934165 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32416
2026-08-12T18:56:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1607 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:56:35 dist-test-slave-rnrw krb5kdc[32528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:56:35 dist-test-slave-rnrw krb5kdc[32528](info): set up 2 sockets
Aug 12 18:56:35 dist-test-slave-rnrw krb5kdc[32528](info): commencing operation
krb5kdc: starting...
W20260812 18:56:36.078464 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:56:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:56:36.228722 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:36.350579 32543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:36.350971 32543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:36.351052 32543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:36.355548 32543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:36.355683 32543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:36.355721 32543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:36.355762 32543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:36.355821 32543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:36.361676 32543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:36.362923 32543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:36.364046 32543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:36.373284 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:36.373284 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:36.373318 32548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:36.373426 32543 server_base.cc:1034] running on GCE node
I20260812 18:56:36.374253 32543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:36.375398 32543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:36.376597 32543 hybrid_clock.cc:648] HybridClock initialized: now 1786560996376532 us; error 34 us; skew 500 ppm
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:36.383212 32543 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:36.384358 32543 webserver.cc:460] Webserver started at http://127.16.40.254:41171/ using document root <none> and password file <none>
I20260812 18:56:36.385089 32543 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:36.385160 32543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:36.385349 32543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:36.387576 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "a706725a45ba4b5982bba101691058a9"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "4ecb5b808400825a15ea261e84f2320b"
server_key_iv: "051ced87b4796ed828677b20f9380479"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.388167 32543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "a706725a45ba4b5982bba101691058a9"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "4ecb5b808400825a15ea261e84f2320b"
server_key_iv: "051ced87b4796ed828677b20f9380479"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.392391 32543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:56:36.395282 32558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:36.396785 32543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:36.396960 32543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a706725a45ba4b5982bba101691058a9"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "4ecb5b808400825a15ea261e84f2320b"
server_key_iv: "051ced87b4796ed828677b20f9380479"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.397094 32543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:36.412810 32543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:36.416712 32543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:36.416952 32543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:36.424698 32543 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38533
I20260812 18:56:36.425081 32618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38533 every 8 connection(s)
I20260812 18:56:36.425953 32543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:36.427627 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32543
I20260812 18:56:36.427742 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:56:36.428115 16547 external_mini_cluster.cc:1428] Setting key 64e171aaae2aa8703fc00c34aed81821
I20260812 18:56:36.429520 32619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:36.435761 32619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap starting.
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:36.438604 32619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:36.439523 32619 log.cc:824] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:36.441972 32619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: No bootstrap required, opened a new log
I20260812 18:56:36.444222 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:36.429755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.16.40.254:38533)
0812 18:56:36.430285 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:36.430301 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:36.430340 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:56:36.431059 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:36.431078 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:36.431450 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:36.431858 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:36.432853 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.433930 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.434891 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.435405 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.435591 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:36.438738 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:36.438748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:36.438770 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:36.438810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:36.440825 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:36.441398 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:36.441406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:36.441423 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:36.441580 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:36.441931 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:36.441934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:36.441936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:36.442462 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:36.442625 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:36.442993 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 18:56:36.443373 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":160,"threads_started":1}
I20260812 18:56:36.446319 32619 raft_consensus.cc:348] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:36.446646 32619 raft_consensus.cc:374] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:36.446709 32619 raft_consensus.cc:729] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a706725a45ba4b5982bba101691058a9, State: Initialized, Role: FOLLOWER
I20260812 18:56:36.447208 32619 consensus_queue.cc:260] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:36.447341 32619 raft_consensus.cc:388] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:36.447450 32619 raft_consensus.cc:482] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:36.447571 32619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:36.448666 32619 raft_consensus.cc:504] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:36.449043 32619 leader_election.cc:302] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a706725a45ba4b5982bba101691058a9; no voters:
I20260812 18:56:36.449424 32619 leader_election.cc:288] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:56:36.449517 32624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:36.449702 32624 raft_consensus.cc:686] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 LEADER]: Becoming Leader. State: Replica: a706725a45ba4b5982bba101691058a9, State: Running, Role: LEADER
I20260812 18:56:36.450014 32624 consensus_queue.cc:237] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:36.450866 32619 sys_catalog.cc:564] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:36.452018 32626 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a706725a45ba4b5982bba101691058a9. Latest consensus state: current_term: 1 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:36.452168 32626 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:36.452644 32634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:36.452616 32625 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:36.452764 32625 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:36.456032 32634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:36.462823 32634 catalog_manager.cc:1347] Generated new cluster ID: a9c809eec734445d81eef3020943925d
I20260812 18:56:36.462919 32634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:36.472328 32634 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:56:36.473414 32634 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:36.481827 32634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Generated new TSK 0
I20260812 18:56:36.482772 32634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:56:36.528283 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:36.652123 32649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:36.652451 32649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:36.652495 32649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:36.656848 32649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:36.656929 32649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:36.656994 32649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:36.662645 32649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:36.663795 32649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:36.664835 32649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:36.676276 32654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:36.676339 32649 server_base.cc:1034] running on GCE node
W20260812 18:56:36.676287 32657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:36.676287 32655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:36.677039 32649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:36.677829 32649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:36.679010 32649 hybrid_clock.cc:648] HybridClock initialized: now 1786560996678992 us; error 34 us; skew 500 ppm
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:36.682128 32649 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:36.683130 32649 webserver.cc:460] Webserver started at http://127.16.40.193:42687/ using document root <none> and password file <none>
I20260812 18:56:36.683704 32649 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:36.683768 32649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:36.683943 32649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:36.686200 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "ac136d01c64b49309be251f25b019fee"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "d0f4795ffb270ca96746c220d24693a8"
server_key_iv: "928039ae984065b1094e480e57382542"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.686767 32649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "ac136d01c64b49309be251f25b019fee"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "d0f4795ffb270ca96746c220d24693a8"
server_key_iv: "928039ae984065b1094e480e57382542"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.690909 32649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:56:36.693717 32666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:36.695207 32649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:36.695348 32649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "ac136d01c64b49309be251f25b019fee"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "d0f4795ffb270ca96746c220d24693a8"
server_key_iv: "928039ae984065b1094e480e57382542"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.695459 32649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:36.730597 32649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:36.734349 32649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:36.734540 32649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:36.735168 32649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:36.736109 32649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:36.736160 32649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:36.736205 32649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:36.736222 32649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:36.746317 32649 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:44485
I20260812 18:56:36.746347 327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:44485 every 8 connection(s)
I20260812 18:56:36.747443 32649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:36.757397 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 32649
I20260812 18:56:36.757552 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:56:36.757931 16547 external_mini_cluster.cc:1428] Setting key fade5375d10d26834d6ce80af86cb982
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:36.763695 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:36.749584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:37825 (local address 127.16.40.254:38533)
0812 18:56:36.749786 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:36.749789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:36.750782 (+ 993us) server_negotiation.cc:407] Connection header received
0812 18:56:36.752386 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:36.752390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:36.752454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:36.752562 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:36.753941 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.754514 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.755738 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.756002 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.756125 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:36.759316 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:36.759319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:36.759321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:36.759348 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:36.761037 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:36.761909 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:36.761915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:36.761918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:36.762027 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:36.762402 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:36.762407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:36.762409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:36.762765 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:36.762880 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:36.763434 (+ 554us) server_negotiation.cc:299] Negotiation successful
0812 18:56:36.763565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 18:56:36.764652 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:36.750012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:37825)
0812 18:56:36.750529 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:36.750657 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:36.752147 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:36.752750 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:36.752758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:36.753210 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:36.753758 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:36.753772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.754669 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.754678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:36.755598 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:36.755608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:36.756182 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:36.756186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:36.756385 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:36.757144 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:36.757171 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:36.759109 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:36.761169 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:36.761179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:36.761202 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:36.761649 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:36.762143 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:36.762146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:36.762147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:36.762304 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:36.762886 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:36.762892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:36.763175 (+ 283us) client_negotiation.cc:769] Sending connection context
0812 18:56:36.763346 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 18:56:36.763649 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260812 18:56:36.766309 328 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:36.766614 328 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:36.767230 328 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:36.769006 32577 ts_manager.cc:194] Registered new tserver with Master: ac136d01c64b49309be251f25b019fee (127.16.40.193:44485)
I20260812 18:56:36.770282 32577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:37825
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:56:36.795933 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:36.916529 333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:36.916981 333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:36.917073 333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:36.921304 333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:36.921422 333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:36.921522 333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:36.927217 333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:36.928496 333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:36.929620 333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:56:36.939911 333 server_base.cc:1034] running on GCE node
W20260812 18:56:36.939792 340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:36.939823 344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:36.939792 339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:36.940717 333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:36.941385 333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:36.942588 333 hybrid_clock.cc:648] HybridClock initialized: now 1786560996942571 us; error 46 us; skew 500 ppm
Aug 12 18:56:36 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:36.945719 333 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:36.946699 333 webserver.cc:460] Webserver started at http://127.16.40.194:46257/ using document root <none> and password file <none>
I20260812 18:56:36.947341 333 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:36.947467 333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:36.947726 333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:36.950024 333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "d82b51a88d87458f8b328bade8c74a69"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "26a470f4f654674418ce4c103162bdc5"
server_key_iv: "5d50462c0b3fd61e41b8193b2069ee8b"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.950651 333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "d82b51a88d87458f8b328bade8c74a69"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "26a470f4f654674418ce4c103162bdc5"
server_key_iv: "5d50462c0b3fd61e41b8193b2069ee8b"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.955219 333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:56:36.958171 352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:36.959611 333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:56:36.959779 333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "d82b51a88d87458f8b328bade8c74a69"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "26a470f4f654674418ce4c103162bdc5"
server_key_iv: "5d50462c0b3fd61e41b8193b2069ee8b"
server_key_version: "encryptionkey@0"
I20260812 18:56:36.959915 333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:36.997468 333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:37.001346 333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:37.001586 333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:37.002259 333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:37.003206 333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:37.003281 333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.003355 333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:37.003407 333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.013236 333 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35263
I20260812 18:56:37.013257 477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35263 every 8 connection(s)
I20260812 18:56:37.014233 333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:37.015034 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 333
I20260812 18:56:37.015180 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:56:37.015470 16547 external_mini_cluster.cc:1428] Setting key 0c8e5adedc7e4d6e32e4663a1b4897ef
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560996, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:56:37.032675 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.016315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:41505 (local address 127.16.40.254:38533)
0812 18:56:37.016483 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.016488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.017707 (+ 1219us) server_negotiation.cc:407] Connection header received
0812 18:56:37.019961 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.019967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.020039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.020184 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:37.021613 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.022504 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.023806 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.024255 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.024397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.027694 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:37.027698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:37.027700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:37.027730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:37.029388 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.030194 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.030197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.030198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.030291 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.031214 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.031217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.031219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.031435 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:37.031521 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.032304 (+ 783us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.032435 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:56:37.033794 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.016883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:41505)
0812 18:56:37.017521 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.017583 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.019629 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.020343 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.020350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.020903 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:37.021434 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.021457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.022657 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.022660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.023663 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.023673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.024457 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.024460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.024706 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.025421 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:37.025440 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:37.027385 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:37.029560 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.029568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.029586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.030061 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.030591 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.030594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.030596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.031068 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.031557 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:37.031568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:37.031997 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.032238 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.032765 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":189,"threads_started":1}
I20260812 18:56:37.035290 478 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:37.035576 478 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:37.036211 478 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:37.037552 32577 ts_manager.cc:194] Registered new tserver with Master: d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263)
I20260812 18:56:37.038192 32577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:41505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:56:37.052368 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:37.175268 483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:37.175580 483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:37.175631 483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:37.180013 483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:37.180094 483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:37.180160 483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:37.186071 483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:37.187314 483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:37.188436 483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:56:37.199064 483 server_base.cc:1034] running on GCE node
W20260812 18:56:37.199014 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:37.199014 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:37.199019 490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:37.199824 483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:37.200647 483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:37.201869 483 hybrid_clock.cc:648] HybridClock initialized: now 1786560997201827 us; error 78 us; skew 500 ppm
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:37.205060 483 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:37.206027 483 webserver.cc:460] Webserver started at http://127.16.40.195:43509/ using document root <none> and password file <none>
I20260812 18:56:37.206609 483 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:37.206672 483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:37.206853 483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:37.209146 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
format_stamp: "Formatted at 2026-08-12 18:56:37 on dist-test-slave-rnrw"
server_key: "141c4027794b0cc16581ce72d37cb121"
server_key_iv: "d07b0cbaa0a4bcd922ca4df170b2baf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:37.209729 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
format_stamp: "Formatted at 2026-08-12 18:56:37 on dist-test-slave-rnrw"
server_key: "141c4027794b0cc16581ce72d37cb121"
server_key_iv: "d07b0cbaa0a4bcd922ca4df170b2baf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:37.213976 483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 18:56:37.216636 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.218025 483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:56:37.218138 483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
format_stamp: "Formatted at 2026-08-12 18:56:37 on dist-test-slave-rnrw"
server_key: "141c4027794b0cc16581ce72d37cb121"
server_key_iv: "d07b0cbaa0a4bcd922ca4df170b2baf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:37.218237 483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:37.245417 483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:37.249204 483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:37.249404 483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:37.250057 483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:37.251169 483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:37.251222 483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.251268 483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:37.251283 483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.261058 483 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39845
I20260812 18:56:37.261078 633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39845 every 8 connection(s)
I20260812 18:56:37.262044 483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:37.271219 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 483
I20260812 18:56:37.271337 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:56:37.271703 16547 external_mini_cluster.cc:1428] Setting key 3e366a0d536126eb4fabe458f9569b0b
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:37.277590 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.264160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:45307 (local address 127.16.40.254:38533)
0812 18:56:37.264325 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.264330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.265398 (+ 1068us) server_negotiation.cc:407] Connection header received
0812 18:56:37.266894 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.266897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.266954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.267066 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:37.268407 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.269032 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.270230 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.270458 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.270549 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.273739 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:37.273748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:37.273750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:37.273776 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:37.275372 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.275936 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.275940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.275942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.276026 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.276341 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.276344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.276345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.276693 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:37.276780 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.277294 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.277407 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:56:37.278419 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.264635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:45307)
0812 18:56:37.265220 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.265276 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.266651 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.267248 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.267255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.267672 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:37.268241 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.268255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.269160 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.269163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.270102 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.270112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.270609 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.270611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.270762 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.271602 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:37.271627 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:37.273556 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:37.275554 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.275560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.275574 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.275841 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.276117 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.276119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.276120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.276258 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.276789 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:37.276795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:37.277048 (+ 253us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.277237 (+ 189us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.277520 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260812 18:56:37.279702 634 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:37.279999 634 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:37.280602 634 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:37.281744 32577 ts_manager.cc:194] Registered new tserver with Master: 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
I20260812 18:56:37.282289 32577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:45307
I20260812 18:56:37.286222 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:56:37.296005 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.288358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.16.40.254:38533)
0812 18:56:37.288513 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.288517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.288743 (+ 226us) server_negotiation.cc:407] Connection header received
0812 18:56:37.288898 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.288907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.288949 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.289036 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:37.289749 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.290282 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.291124 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.291341 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.291452 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.292953 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:37.292957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:37.292959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:37.292991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:37.294290 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.294718 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.294722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.294724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.294806 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.295082 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.295085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.295087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.295322 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:37.295466 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.295711 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.295830 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:56:37.299263 32577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:56:37.302592 32577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:56:37.318388 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.312815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.16.40.195:39845)
0812 18:56:37.313355 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.313365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.313384 (+ 19us) server_negotiation.cc:407] Connection header received
0812 18:56:37.313482 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.313488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.313697 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.314296 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:37.314455 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315616 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.316845 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.317643 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.317906 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.318069 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.318123 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.318219 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":190,"threads_started":1}
I20260812 18:56:37.318374 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.312077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39845 (local address 127.0.0.1:42090)
0812 18:56:37.312787 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.312801 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.312895 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.313884 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.313905 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.313933 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:37.314130 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.314139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315803 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.315807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.316696 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.316707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.317792 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.317795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.317933 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.317977 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.318090 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.318230 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":18,"thread_start_us":101,"threads_started":1}
I20260812 18:56:37.318564 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.312050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35263 (local address 127.0.0.1:55426)
0812 18:56:37.312425 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.312475 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.312630 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.314047 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.314049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.314068 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:37.314284 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.314292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315764 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.315768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.316993 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.317003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.318105 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.318109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.318207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.318230 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.318314 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.318393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":120,"thread_start_us":112,"threads_started":1}
I20260812 18:56:37.319033 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.312183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.16.40.194:35263)
0812 18:56:37.313491 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.313513 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.313538 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:56:37.313616 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.313621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.313811 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.314140 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:37.314422 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315611 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.317130 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.317979 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.318102 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.318237 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.318343 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.318453 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":109,"threads_started":1}
I20260812 18:56:37.319244 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.312876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44485 (local address 127.0.0.1:53072)
0812 18:56:37.313107 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.313121 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.313207 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.314113 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.314116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.314128 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:37.314248 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.314256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315268 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.315275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.316532 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.316585 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.317758 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.317765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.318912 (+ 1147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.318932 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.319030 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.319104 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":24,"thread_start_us":98,"threads_started":1}
I20260812 18:56:37.319479 655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.313050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.16.40.193:44485)
0812 18:56:37.313568 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.313573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.313590 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:56:37.313650 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.313654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.313810 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.313973 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:37.314385 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.315149 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.316730 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.317611 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.317726 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.317885 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.319069 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.319268 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":96,"threads_started":1}
I20260812 18:56:37.321869 406 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9a34efeab45049d682636a7aba8b1 (DEFAULT_TABLE table=test-table [id=3f995093a49d49af892854020a9058e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:37.321869 557 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9a34efeab45049d682636a7aba8b1 (DEFAULT_TABLE table=test-table [id=3f995093a49d49af892854020a9058e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:37.322917 406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9a34efeab45049d682636a7aba8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:37.322956 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9a34efeab45049d682636a7aba8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:37.327859 32720 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9a34efeab45049d682636a7aba8b1 (DEFAULT_TABLE table=test-table [id=3f995093a49d49af892854020a9058e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:37.329571 657 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Bootstrap starting.
I20260812 18:56:37.330744 32720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9a34efeab45049d682636a7aba8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:37.332300 657 tablet_bootstrap.cc:654] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:37.332922 658 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Bootstrap starting.
I20260812 18:56:37.333721 657 log.cc:824] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:37.335704 658 tablet_bootstrap.cc:654] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:37.336215 657 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: No bootstrap required, opened a new log
I20260812 18:56:37.336434 657 ts_tablet_manager.cc:1397] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260812 18:56:37.336673 658 log.cc:824] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:37.338474 658 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: No bootstrap required, opened a new log
I20260812 18:56:37.338689 658 ts_tablet_manager.cc:1397] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:56:37.338922 661 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Bootstrap starting.
I20260812 18:56:37.340200 657 raft_consensus.cc:348] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.340471 657 raft_consensus.cc:374] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:37.340567 657 raft_consensus.cc:729] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bd3d3b8749483e9a5f0e5c0f6b0a41, State: Initialized, Role: FOLLOWER
I20260812 18:56:37.341053 657 consensus_queue.cc:260] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.341466 661 tablet_bootstrap.cc:654] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:37.342007 657 ts_tablet_manager.cc:1428] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260812 18:56:37.342397 634 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
I20260812 18:56:37.342670 658 raft_consensus.cc:348] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.342926 658 raft_consensus.cc:374] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:37.343024 658 raft_consensus.cc:729] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82b51a88d87458f8b328bade8c74a69, State: Initialized, Role: FOLLOWER
I20260812 18:56:37.343470 658 consensus_queue.cc:260] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.343751 661 log.cc:824] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:37.344195 478 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
I20260812 18:56:37.344568 658 ts_tablet_manager.cc:1428] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:56:37.346267 661 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: No bootstrap required, opened a new log
I20260812 18:56:37.346511 661 ts_tablet_manager.cc:1397] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 18:56:37.350490 661 raft_consensus.cc:348] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.350782 661 raft_consensus.cc:374] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:37.350876 661 raft_consensus.cc:729] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac136d01c64b49309be251f25b019fee, State: Initialized, Role: FOLLOWER
I20260812 18:56:37.351462 661 consensus_queue.cc:260] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.352241 328 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
I20260812 18:56:37.352677 661 ts_tablet_manager.cc:1428] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260812 18:56:37.413012 665 raft_consensus.cc:482] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:37.413281 665 raft_consensus.cc:504] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.414480 665 leader_election.cc:288] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
I20260812 18:56:37.419687 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.414922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:50093 (local address 127.16.40.194:35263)
0812 18:56:37.415067 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.415072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.415372 (+ 300us) server_negotiation.cc:407] Connection header received
0812 18:56:37.415543 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.415546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.415599 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.415684 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:37.416015 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.416519 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.418338 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.418974 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.419107 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.419174 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.419468 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.419543 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 18:56:37.419687 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.415036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:54369 (local address 127.16.40.195:39845)
0812 18:56:37.415207 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.415212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.415436 (+ 224us) server_negotiation.cc:407] Connection header received
0812 18:56:37.415585 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.415588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.415640 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.415723 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:37.416025 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.416520 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.418338 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.418969 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.419125 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.419174 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.419468 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.419543 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 18:56:37.419772 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.414815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35263 (local address 127.16.40.193:50093)
0812 18:56:37.415269 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.415282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.415418 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.415698 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.415701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.415716 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:37.415898 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.415903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.416685 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.416687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.418199 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.418210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.419107 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.419136 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.419286 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.419356 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.419437 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.419575 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":172,"spinlock_wait_cycles":30976,"thread_start_us":146,"threads_started":1}
I20260812 18:56:37.419912 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.414815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39845 (local address 127.16.40.193:54369)
0812 18:56:37.415334 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.415347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.415480 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.415745 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.415748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.415792 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:37.415917 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.415922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.416677 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.416681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.418198 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.418210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.419107 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.419111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.419296 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.419356 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.419438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.419575 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":286,"spinlock_wait_cycles":4224,"thread_start_us":115,"threads_started":1}
I20260812 18:56:37.420887 427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e9a34efeab45049d682636a7aba8b1" candidate_uuid: "ac136d01c64b49309be251f25b019fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82b51a88d87458f8b328bade8c74a69" is_pre_election: true
I20260812 18:56:37.420966 579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e9a34efeab45049d682636a7aba8b1" candidate_uuid: "ac136d01c64b49309be251f25b019fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" is_pre_election: true
I20260812 18:56:37.421289 579 raft_consensus.cc:2454] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac136d01c64b49309be251f25b019fee in term 0.
I20260812 18:56:37.421289 427 raft_consensus.cc:2454] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac136d01c64b49309be251f25b019fee in term 0.
I20260812 18:56:37.421877 32670 leader_election.cc:302] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, ac136d01c64b49309be251f25b019fee, d82b51a88d87458f8b328bade8c74a69; no voters:
I20260812 18:56:37.422245 665 raft_consensus.cc:2781] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:56:37.422376 665 raft_consensus.cc:482] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:56:37.422446 665 raft_consensus.cc:3037] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:37.423606 665 raft_consensus.cc:504] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.423997 665 leader_election.cc:288] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 1 election: Requested vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
I20260812 18:56:37.424500 427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e9a34efeab45049d682636a7aba8b1" candidate_uuid: "ac136d01c64b49309be251f25b019fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82b51a88d87458f8b328bade8c74a69"
I20260812 18:56:37.424597 579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52e9a34efeab45049d682636a7aba8b1" candidate_uuid: "ac136d01c64b49309be251f25b019fee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
I20260812 18:56:37.424679 427 raft_consensus.cc:3037] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:37.424722 579 raft_consensus.cc:3037] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:37.426189 579 raft_consensus.cc:2454] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac136d01c64b49309be251f25b019fee in term 1.
I20260812 18:56:37.426189 427 raft_consensus.cc:2454] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac136d01c64b49309be251f25b019fee in term 1.
I20260812 18:56:37.426564 32670 leader_election.cc:302] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, ac136d01c64b49309be251f25b019fee; no voters:
I20260812 18:56:37.426892 665 raft_consensus.cc:2781] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:37.427147 665 raft_consensus.cc:686] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 LEADER]: Becoming Leader. State: Replica: ac136d01c64b49309be251f25b019fee, State: Running, Role: LEADER
I20260812 18:56:37.427515 665 consensus_queue.cc:237] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:37.431226 32576 catalog_manager.cc:5495] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee reported cstate change: term changed from 0 to 1, leader changed from <none> to ac136d01c64b49309be251f25b019fee (127.16.40.193). New cstate: current_term: 1 leader_uuid: "ac136d01c64b49309be251f25b019fee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } health_report { overall_health: HEALTHY } } }
I20260812 18:56:37.445930 655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.442001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.16.40.193:44485)
0812 18:56:37.442169 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.442172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.442187 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:56:37.442282 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.442284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.442347 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.442448 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:56:37.442695 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.443357 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.444263 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.444486 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.444707 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.444997 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:56:37.445465 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:56:37.445569 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.445683 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.445749 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 18:56:37.453145 427 raft_consensus.cc:1261] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Refusing update from remote peer ac136d01c64b49309be251f25b019fee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:37.453167 579 raft_consensus.cc:1261] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Refusing update from remote peer ac136d01c64b49309be251f25b019fee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:37.454007 665 consensus_queue.cc:1035] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:37.454275 668 consensus_queue.cc:1035] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:37.461329 672 mvcc.cc:205] Tried to move back new op lower bound from 7317753845558472704 to 7317753845466177536. Current Snapshot: MvccSnapshot[applied={T|T < 7317753845558472704}]
I20260812 18:56:37.463891 674 mvcc.cc:205] Tried to move back new op lower bound from 7317753845558472704 to 7317753845466177536. Current Snapshot: MvccSnapshot[applied={T|T < 7317753845558472704}]
I20260812 18:56:37.464493 676 mvcc.cc:205] Tried to move back new op lower bound from 7317753845558472704 to 7317753845466177536. Current Snapshot: MvccSnapshot[applied={T|T < 7317753845558472704}]
I20260812 18:56:37.472414 32575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:37.472764 32575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:37.475546 32575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Sending DeleteTablet for 3 replicas of tablet 52e9a34efeab45049d682636a7aba8b1
I20260812 18:56:37.476711 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32649
I20260812 18:56:37.476631 557 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9a34efeab45049d682636a7aba8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:37 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:42090
I20260812 18:56:37.476641 32720 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9a34efeab45049d682636a7aba8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:37 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:53072
I20260812 18:56:37.476678 406 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9a34efeab45049d682636a7aba8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:37 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260812 18:56:37.479423 685 tablet_replica.cc:330] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: stopping tablet replica
I20260812 18:56:37.479828 685 raft_consensus.cc:2229] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:37.480213 685 raft_consensus.cc:2258] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:37.481428 687 tablet_replica.cc:330] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: stopping tablet replica
I20260812 18:56:37.481786 687 raft_consensus.cc:2229] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:37.482122 687 raft_consensus.cc:2258] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:37.482213 685 ts_tablet_manager.cc:1905] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:37.483784 687 ts_tablet_manager.cc:1905] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:37.485298 685 ts_tablet_manager.cc:1918] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:37.485423 685 log.cc:1196] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/52e9a34efeab45049d682636a7aba8b1
I20260812 18:56:37.485785 685 ts_tablet_manager.cc:1939] T 52e9a34efeab45049d682636a7aba8b1 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting consensus metadata
I20260812 18:56:37.486711 687 ts_tablet_manager.cc:1918] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:37.486825 687 log.cc:1196] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/52e9a34efeab45049d682636a7aba8b1
I20260812 18:56:37.487167 687 ts_tablet_manager.cc:1939] T 52e9a34efeab45049d682636a7aba8b1 P d82b51a88d87458f8b328bade8c74a69: Deleting consensus metadata
I20260812 18:56:37.487663 32563 catalog_manager.cc:4841] TS 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): tablet 52e9a34efeab45049d682636a7aba8b1 (table test-table [id=3f995093a49d49af892854020a9058e1]) successfully deleted
I20260812 18:56:37.488684 32562 catalog_manager.cc:4841] TS d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): tablet 52e9a34efeab45049d682636a7aba8b1 (table test-table [id=3f995093a49d49af892854020a9058e1]) successfully deleted
W20260812 18:56:37.489586 32562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:44485 (error 108)
I20260812 18:56:37.489890 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 333
I20260812 18:56:37.491060 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.490717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44485 (local address 127.0.0.1:53100)
0812 18:56:37.490863 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.490952 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 18:56:37.491449 32562 catalog_manager.cc:4584] TS ac136d01c64b49309be251f25b019fee (127.16.40.193:44485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52e9a34efeab45049d682636a7aba8b1: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44485: connect: Connection refused (error 111)
I20260812 18:56:37.498972 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 483
I20260812 18:56:37.512769 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 32543
I20260812 18:56:37.524935 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--webserver_interface=127.16.40.254
--webserver_port=41171
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:37.645326 689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:37.645641 689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:37.645685 689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:37.649937 689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:37.650024 689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:37.650045 689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:37.650063 689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:37.650120 689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:37.655934 689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:37.657222 689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:37.658356 689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:37.673556 697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:37.673557 696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:37.673715 689 server_base.cc:1034] running on GCE node
W20260812 18:56:37.673889 699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:37.674518 689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:37.675899 689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:37.677124 689 hybrid_clock.cc:648] HybridClock initialized: now 1786560997677105 us; error 36 us; skew 500 ppm
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:37.680163 689 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:37.681190 689 webserver.cc:460] Webserver started at http://127.16.40.254:41171/ using document root <none> and password file <none>
I20260812 18:56:37.681780 689 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:37.681854 689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:37.686007 689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:37.688607 710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.689997 689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:37.690115 689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a706725a45ba4b5982bba101691058a9"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "4ecb5b808400825a15ea261e84f2320b"
server_key_iv: "051ced87b4796ed828677b20f9380479"
server_key_version: "encryptionkey@0"
I20260812 18:56:37.690781 689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:37.705922 689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:37.709658 689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:37.709851 689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:37.717831 689 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38533
I20260812 18:56:37.718160 774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38533 every 8 connection(s)
I20260812 18:56:37.719241 689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:37.723572 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 689
I20260812 18:56:37.724130 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:44485
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=42687
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:37.725118 775 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:56:37.727641 775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap starting.
I20260812 18:56:37.753873 775 log.cc:824] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:37.766144 775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:56:37.766670 775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap complete.
I20260812 18:56:37.772822 775 raft_consensus.cc:348] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:37.773404 775 raft_consensus.cc:729] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a706725a45ba4b5982bba101691058a9, State: Initialized, Role: FOLLOWER
I20260812 18:56:37.774039 775 consensus_queue.cc:260] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:37.774202 775 raft_consensus.cc:388] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:37.774286 775 raft_consensus.cc:482] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:37.774395 775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:56:37.777384 775 raft_consensus.cc:504] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:37.777890 775 leader_election.cc:302] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a706725a45ba4b5982bba101691058a9; no voters:
I20260812 18:56:37.778330 775 leader_election.cc:288] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 18:56:37.778493 779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 18:56:37.778796 779 raft_consensus.cc:686] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 LEADER]: Becoming Leader. State: Replica: a706725a45ba4b5982bba101691058a9, State: Running, Role: LEADER
I20260812 18:56:37.779170 779 consensus_queue.cc:237] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:37.779815 775 sys_catalog.cc:564] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:37.780916 780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:37.781049 780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:37.780889 782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a706725a45ba4b5982bba101691058a9. Latest consensus state: current_term: 2 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:37.781179 782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:37.781481 792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:37.787356 792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:37.788365 792 catalog_manager.cc:1259] Loaded cluster ID: a9c809eec734445d81eef3020943925d
I20260812 18:56:37.788437 792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:37.789969 792 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 18:56:37.794981 792 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 18:56:37.801203 792 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:37.802141 792 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Loaded TSK: 0
I20260812 18:56:37.803449 792 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:56:37.869577 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:37.869900 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:37.869992 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:37.874444 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:37.874562 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:37.874666 777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:37.880599 777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:44485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=42687
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:37.881987 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:37.883203 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:37.895080 810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:37.895138 808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:37.895548 806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:37.895735 777 server_base.cc:1034] running on GCE node
I20260812 18:56:37.896270 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:37.897229 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:37.898450 777 hybrid_clock.cc:648] HybridClock initialized: now 1786560997898422 us; error 40 us; skew 500 ppm
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:37.902460 777 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:37.903812 777 webserver.cc:460] Webserver started at http://127.16.40.193:42687/ using document root <none> and password file <none>
I20260812 18:56:37.904687 777 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:37.904814 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:37.910593 777 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:56:37.913894 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:37.915673 777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:37.915894 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "ac136d01c64b49309be251f25b019fee"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "d0f4795ffb270ca96746c220d24693a8"
server_key_iv: "928039ae984065b1094e480e57382542"
server_key_version: "encryptionkey@0"
I20260812 18:56:37.916630 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:37.951435 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:37.955608 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:37.955868 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:37.956660 777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:37.957978 828 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 18:56:37.963130 777 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 18:56:37.963270 777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 18:56:37.963371 777 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 18:56:37.965569 777 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 18:56:37.965663 777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260812 18:56:37.965967 828 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Bootstrap starting.
I20260812 18:56:37.977941 777 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:44485
I20260812 18:56:37.978080 950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:44485 every 8 connection(s)
I20260812 18:56:37.979164 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:37.984627 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 777
I20260812 18:56:37.985176 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:35263
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=46257
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:37.987464 828 log.cc:824] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:56:37 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:37.998818 828 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:56:37.999396 828 tablet_bootstrap.cc:492] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Bootstrap complete.
I20260812 18:56:38.000257 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.982142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:49957)
0812 18:56:37.982706 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:37.982755 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:37.983648 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:37.985474 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:37.985486 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:37.985889 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:37.987585 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.987619 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.989760 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.989763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.990697 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:37.990708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.991621 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.991627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:37.991815 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.992527 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:37.992592 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:37.994191 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:37.996780 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.996787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.996802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.997103 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.997525 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:37.997529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:37.997531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:37.997698 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:37.998361 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:37.998369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:37.998637 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 18:56:37.998832 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 18:56:37.999218 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":168,"threads_started":1}
I20260812 18:56:38.000238 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:37.982020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:49957 (local address 127.16.40.254:38533)
0812 18:56:37.982439 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:37.982465 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:37.982901 (+ 436us) server_negotiation.cc:407] Connection header received
0812 18:56:37.984122 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:37.984136 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:37.984494 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:37.984947 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:37.988361 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.989554 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.990854 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:37.991446 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:37.991666 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:37.994491 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:37.994501 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:37.994525 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:37.994560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:37.996625 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.997244 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.997253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.997263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.997402 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:37.997813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:37.997817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:37.997819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:37.998225 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:37.998378 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:37.999107 (+ 729us) server_negotiation.cc:299] Negotiation successful
0812 18:56:37.999422 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":159,"threads_started":1}
I20260812 18:56:38.001112 828 ts_tablet_manager.cc:1397] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.009s
I20260812 18:56:38.003157 951 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:38.003439 951 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:38.004086 951 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:38.004328 828 raft_consensus.cc:348] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.004927 828 raft_consensus.cc:729] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac136d01c64b49309be251f25b019fee, State: Initialized, Role: FOLLOWER
I20260812 18:56:38.005499 828 consensus_queue.cc:260] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.008709 828 ts_tablet_manager.cc:1428] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 18:56:38.009424 731 ts_manager.cc:194] Registered new tserver with Master: ac136d01c64b49309be251f25b019fee (127.16.40.193:44485)
I20260812 18:56:38.013375 731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:49957
I20260812 18:56:38.016316 951 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560997, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:56:38.025632 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.012844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44485 (local address 127.0.0.1:53114)
0812 18:56:38.013849 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.013898 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.014073 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.014796 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.014802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.015286 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:38.015429 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.015438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.016868 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.016875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.017762 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.017779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.018592 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.018594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.018684 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.019292 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:38.019305 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:38.020962 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:38.023450 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.023461 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.023468 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.023773 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.024171 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.024175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.024177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.024283 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.024934 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:38.024940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:38.025083 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.025239 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.025459 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":166,"threads_started":1}
I20260812 18:56:38.025712 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.012973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.16.40.193:44485)
0812 18:56:38.013441 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.013454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.014053 (+ 599us) server_negotiation.cc:407] Connection header received
0812 18:56:38.014257 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.014263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.014524 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.014680 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:38.015589 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.016750 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.017912 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.018459 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.018580 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.021138 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:38.021152 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:38.021167 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:38.021199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:38.023270 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.023908 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.023915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.023917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.024039 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.024390 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.024393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.024395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.024782 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:38.024925 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.025339 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.025538 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":120,"threads_started":1}
I20260812 18:56:38.026963 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9a34efeab45049d682636a7aba8b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:37 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:53114
I20260812 18:56:38.027963 963 tablet_replica.cc:330] stopping tablet replica
I20260812 18:56:38.028302 963 raft_consensus.cc:2229] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:38.028630 963 raft_consensus.cc:2258] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:38.029706 963 ts_tablet_manager.cc:1905] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:38.033619 963 ts_tablet_manager.cc:1918] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:38.033764 963 log.cc:1196] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/52e9a34efeab45049d682636a7aba8b1
I20260812 18:56:38.034050 963 ts_tablet_manager.cc:1939] T 52e9a34efeab45049d682636a7aba8b1 P ac136d01c64b49309be251f25b019fee: Deleting consensus metadata
I20260812 18:56:38.035940 713 catalog_manager.cc:4841] TS ac136d01c64b49309be251f25b019fee (127.16.40.193:44485): tablet 52e9a34efeab45049d682636a7aba8b1 (table test-table [id=3f995093a49d49af892854020a9058e1]) successfully deleted
W20260812 18:56:38.124217 956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:38.124644 956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:38.124758 956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:38.129164 956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:38.129276 956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:38.129374 956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:38.135443 956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:35263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=46257
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:38.136801 956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:38.137941 956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:38.148844 972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:38.148869 969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:38.149013 956 server_base.cc:1034] running on GCE node
W20260812 18:56:38.148844 970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:38.149788 956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:38.150552 956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:38.151743 956 hybrid_clock.cc:648] HybridClock initialized: now 1786560998151706 us; error 49 us; skew 500 ppm
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560998, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:38.155037 956 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:38.156123 956 webserver.cc:460] Webserver started at http://127.16.40.194:46257/ using document root <none> and password file <none>
I20260812 18:56:38.156859 956 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:38.156993 956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:38.161363 956 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:38.164000 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.165512 956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:56:38.165693 956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "d82b51a88d87458f8b328bade8c74a69"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "26a470f4f654674418ce4c103162bdc5"
server_key_iv: "5d50462c0b3fd61e41b8193b2069ee8b"
server_key_version: "encryptionkey@0"
I20260812 18:56:38.166317 956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:38.180497 956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:38.184413 956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:38.184696 956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:38.185448 956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:38.186421 956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:38.186499 956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.186617 956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:38.186657 956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.197299 956 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35263
I20260812 18:56:38.197340 1113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35263 every 8 connection(s)
I20260812 18:56:38.198459 956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:38.205425 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 956
I20260812 18:56:38.205899 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:39845
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=43509
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560998, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:38.215605 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.200725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:36931 (local address 127.16.40.254:38533)
0812 18:56:38.200881 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.200885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.202002 (+ 1117us) server_negotiation.cc:407] Connection header received
0812 18:56:38.203448 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.203452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.203523 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.203610 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:38.205046 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.205690 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.207026 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.207288 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.207434 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.210661 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:38.210670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:38.210676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:38.210718 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:38.212901 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.213599 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.213606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.213610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.213727 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.214145 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.214152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.214156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.214457 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:38.214578 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.215178 (+ 600us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.215362 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:56:38.216369 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.201216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:36931)
0812 18:56:38.201808 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.201854 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.203215 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.203800 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.203807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.204217 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:38.204883 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.204898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.205837 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.205840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.206889 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.206900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.207496 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.207500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.207679 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.208372 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:38.208391 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:38.210329 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:38.213096 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.213105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.213123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.213463 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.213840 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.213842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.213843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.214026 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.214585 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:38.214592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:38.214889 (+ 297us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.215061 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.215360 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":139,"threads_started":1}
I20260812 18:56:38.218127 1114 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:38.218466 1114 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:38.219235 1114 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:38.220729 731 ts_manager.cc:194] Registered new tserver with Master: d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263)
I20260812 18:56:38.221519 731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:36931
W20260812 18:56:38.332796 1118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:38.333117 1118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:38.333191 1118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:38.337591 1118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:38.337682 1118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:38.337746 1118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:38.343535 1118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=43509
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:38.345072 1118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:38.346349 1118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:56:38.357836 1118 server_base.cc:1034] running on GCE node
W20260812 18:56:38.357800 1126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:38.357816 1124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:38.358083 1123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:38.358592 1118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:38.359395 1118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:38.360633 1118 hybrid_clock.cc:648] HybridClock initialized: now 1786560998360591 us; error 61 us; skew 500 ppm
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560998, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:38.363780 1118 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:38.364969 1118 webserver.cc:460] Webserver started at http://127.16.40.195:43509/ using document root <none> and password file <none>
I20260812 18:56:38.365631 1118 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:38.365720 1118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:38.370455 1118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:38.373188 1133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.374818 1118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:38.375005 1118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
format_stamp: "Formatted at 2026-08-12 18:56:37 on dist-test-slave-rnrw"
server_key: "141c4027794b0cc16581ce72d37cb121"
server_key_iv: "d07b0cbaa0a4bcd922ca4df170b2baf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:38.375670 1118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:38.394651 1118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:38.398491 1118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:38.398680 1118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:38.399312 1118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:38.400220 1118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:38.400271 1118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.400315 1118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:38.400331 1118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.411356 1118 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39845
I20260812 18:56:38.411378 1262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39845 every 8 connection(s)
I20260812 18:56:38.412514 1118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:38.415380 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1118
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560998, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:38.425930 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.416248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.16.40.254:38533)
0812 18:56:38.416497 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.416502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.416714 (+ 212us) server_negotiation.cc:407] Connection header received
0812 18:56:38.416893 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.416897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.416954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.417059 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:38.417858 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.418525 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.419458 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.419790 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.419903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.422024 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:38.422028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:38.422030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:38.422066 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:38.423830 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.424433 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.424436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.424438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.424530 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.424933 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.424938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.424940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.425175 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:38.425418 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.425568 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.425725 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260812 18:56:38.430965 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.414764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:45211 (local address 127.16.40.254:38533)
0812 18:56:38.414934 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.414939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.416292 (+ 1353us) server_negotiation.cc:407] Connection header received
0812 18:56:38.417983 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.417985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.418032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.418116 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:38.419645 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.420250 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.422243 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.422602 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.422758 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.426003 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:38.426009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:38.426011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:38.426053 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:38.427977 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.428826 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.428829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.428831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.428919 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:38.429401 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:38.429404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:38.429406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:38.429658 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:38.429761 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.430675 (+ 914us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.430818 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:56:38.431876 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.415363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:45211)
0812 18:56:38.416087 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.416162 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.417778 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.418350 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.418358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.418892 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:38.419493 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.419509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.420454 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.420457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.422113 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.422126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.422855 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.422859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.423104 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.423745 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:38.423764 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:38.425783 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:38.428175 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.428186 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.428210 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.428708 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.429050 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:38.429053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:38.429054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:38.429238 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:38.429806 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:38.429812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:38.430276 (+ 464us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.430536 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.430869 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":177,"threads_started":1}
I20260812 18:56:38.433290 1263 heartbeater.cc:347] Connected to a master server at 127.16.40.254:38533
I20260812 18:56:38.433550 1263 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:38.434075 1263 heartbeater.cc:511] Master 127.16.40.254:38533 requested a full tablet report, sending...
I20260812 18:56:38.435322 731 ts_manager.cc:194] Registered new tserver with Master: 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
I20260812 18:56:38.435930 731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:45211
I20260812 18:56:38.440500 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:56:38.445390 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.441285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.16.40.254:38533)
0812 18:56:38.441454 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.441459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.441678 (+ 219us) server_negotiation.cc:407] Connection header received
0812 18:56:38.441829 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.441831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.441950 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.442067 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:56:38.442298 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.442842 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.443763 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.443993 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.444086 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.444391 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:56:38.444941 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:56:38.445068 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.445149 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.445209 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 18:56:38.446223 731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:56:38.449913 731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:56:38.460158 879 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 (DEFAULT_TABLE table=test-table [id=c017fb5814904b879822d9872859cfbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:38.460945 879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff63627285b49938c4eaa9d6f8541c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:38.464767 828 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Bootstrap starting.
I20260812 18:56:38.464851 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.458337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35263 (local address 127.0.0.1:34702)
0812 18:56:38.458489 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.458507 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.458601 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.459639 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.459643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.459666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:38.459786 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.459793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.461386 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.461391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.462655 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.462670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.464273 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.464277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.464378 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.464438 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.464527 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.464655 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 18:56:38.464860 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.458588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.16.40.194:35263)
0812 18:56:38.459004 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.459019 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.459045 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:56:38.459166 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.459172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.459437 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.459628 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:38.459939 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.461226 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.462847 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.464133 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.464272 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.464445 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.464596 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.464709 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":171,"threads_started":1}
I20260812 18:56:38.464883 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.458586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39845 (local address 127.0.0.1:41574)
0812 18:56:38.458924 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.458938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.459030 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.460042 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.460045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.460066 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:38.460179 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.460193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.462044 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.462048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.463190 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.463202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.464400 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.464406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.464496 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.464524 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.464612 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.464703 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":152,"threads_started":1}
I20260812 18:56:38.465874 828 tablet_bootstrap.cc:654] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:38.465915 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.459074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.16.40.195:39845)
0812 18:56:38.459467 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.459478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.459507 (+ 29us) server_negotiation.cc:407] Connection header received
0812 18:56:38.459651 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.459657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.459873 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.460607 (+ 734us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:38.460796 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.461864 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.463365 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.464234 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.465353 (+ 1119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.465530 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.465589 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.465694 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":166,"threads_started":1}
I20260812 18:56:38.468264 828 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: No bootstrap required, opened a new log
I20260812 18:56:38.468394 828 ts_tablet_manager.cc:1397] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 18:56:38.468937 828 raft_consensus.cc:348] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.468516 1039 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 (DEFAULT_TABLE table=test-table [id=c017fb5814904b879822d9872859cfbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:38.469074 828 raft_consensus.cc:374] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:38.469132 828 raft_consensus.cc:729] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac136d01c64b49309be251f25b019fee, State: Initialized, Role: FOLLOWER
I20260812 18:56:38.468931 1188 tablet_service.cc:1467] Processing CreateTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 (DEFAULT_TABLE table=test-table [id=c017fb5814904b879822d9872859cfbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:38.469304 828 consensus_queue.cc:260] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.469805 1039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff63627285b49938c4eaa9d6f8541c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:38.469895 1188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ff63627285b49938c4eaa9d6f8541c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:38.470494 828 ts_tablet_manager.cc:1428] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 18:56:38.479142 1278 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Bootstrap starting.
I20260812 18:56:38.479142 1279 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Bootstrap starting.
W20260812 18:56:38.480720 952 tablet.cc:2369] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:56:38.482432 1279 tablet_bootstrap.cc:654] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:38.482929 1278 tablet_bootstrap.cc:654] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:38.483981 1278 log.cc:824] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:38.483989 1279 log.cc:824] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:38.486176 1279 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: No bootstrap required, opened a new log
I20260812 18:56:38.486177 1278 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: No bootstrap required, opened a new log
I20260812 18:56:38.486431 1279 ts_tablet_manager.cc:1397] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260812 18:56:38.486460 1278 ts_tablet_manager.cc:1397] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260812 18:56:38.489745 1279 raft_consensus.cc:348] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.489741 1278 raft_consensus.cc:348] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.490082 1279 raft_consensus.cc:374] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:38.490083 1278 raft_consensus.cc:374] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:38.490145 1279 raft_consensus.cc:729] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bd3d3b8749483e9a5f0e5c0f6b0a41, State: Initialized, Role: FOLLOWER
I20260812 18:56:38.490150 1278 raft_consensus.cc:729] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82b51a88d87458f8b328bade8c74a69, State: Initialized, Role: FOLLOWER
I20260812 18:56:38.490674 1278 consensus_queue.cc:260] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.490717 1279 consensus_queue.cc:260] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.491501 1114 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
I20260812 18:56:38.491719 1263 heartbeater.cc:503] Master 127.16.40.254:38533 was elected leader, sending a full tablet report...
I20260812 18:56:38.491806 1279 ts_tablet_manager.cc:1428] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 18:56:38.491801 1278 ts_tablet_manager.cc:1428] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 18:56:38.505023 1282 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:38.505307 1282 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.506680 1282 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845), ac136d01c64b49309be251f25b019fee (127.16.40.193:44485)
I20260812 18:56:38.511152 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.507060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:45981 (local address 127.16.40.195:39845)
0812 18:56:38.507235 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.507243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.507263 (+ 20us) server_negotiation.cc:407] Connection header received
0812 18:56:38.507330 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.507334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.507388 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.507527 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:38.507815 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.508325 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.509934 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.510481 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.510614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.510678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.510903 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.511011 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 18:56:38.511174 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.506899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39845 (local address 127.16.40.194:45981)
0812 18:56:38.507072 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.507090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.507200 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.507539 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.507542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.507563 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:38.507691 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.507697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.508447 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.508450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.509763 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.509775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.510663 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.510667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.510763 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.510788 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.510862 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.510927 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1}
I20260812 18:56:38.511731 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.507100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44485 (local address 127.16.40.194:47991)
0812 18:56:38.507520 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.507531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:38.507652 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:38.507982 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:38.507985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:38.508003 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:38.508124 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.508129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.509011 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.509014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.510106 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:38.510121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.511115 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.511119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:38.511391 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.511428 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 18:56:38.511512 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:56:38.511588 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":23,"thread_start_us":161,"threads_started":1}
I20260812 18:56:38.511799 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.507212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:47991 (local address 127.16.40.193:44485)
0812 18:56:38.507347 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.507353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.507626 (+ 273us) server_negotiation.cc:407] Connection header received
0812 18:56:38.507775 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.507783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.507842 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.507952 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:38.508271 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.508864 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.510229 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.510921 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.511052 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.511238 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.511543 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.511658 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:56:38.512122 1212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff63627285b49938c4eaa9d6f8541c0" candidate_uuid: "d82b51a88d87458f8b328bade8c74a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" is_pre_election: true
I20260812 18:56:38.512403 899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff63627285b49938c4eaa9d6f8541c0" candidate_uuid: "d82b51a88d87458f8b328bade8c74a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac136d01c64b49309be251f25b019fee" is_pre_election: true
I20260812 18:56:38.512490 1212 raft_consensus.cc:2454] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d82b51a88d87458f8b328bade8c74a69 in term 0.
I20260812 18:56:38.512730 899 raft_consensus.cc:2454] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d82b51a88d87458f8b328bade8c74a69 in term 0.
I20260812 18:56:38.513120 984 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, d82b51a88d87458f8b328bade8c74a69; no voters:
I20260812 18:56:38.513489 1282 raft_consensus.cc:2781] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:56:38.513640 1282 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:56:38.513726 1282 raft_consensus.cc:3037] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:38.514899 1282 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.515385 1282 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [CANDIDATE]: Term 1 election: Requested vote from peers 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845), ac136d01c64b49309be251f25b019fee (127.16.40.193:44485)
I20260812 18:56:38.515918 1212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff63627285b49938c4eaa9d6f8541c0" candidate_uuid: "d82b51a88d87458f8b328bade8c74a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
I20260812 18:56:38.515945 899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ff63627285b49938c4eaa9d6f8541c0" candidate_uuid: "d82b51a88d87458f8b328bade8c74a69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac136d01c64b49309be251f25b019fee"
I20260812 18:56:38.516088 1212 raft_consensus.cc:3037] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:38.516112 899 raft_consensus.cc:3037] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:38.517403 1212 raft_consensus.cc:2454] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d82b51a88d87458f8b328bade8c74a69 in term 1.
I20260812 18:56:38.517409 899 raft_consensus.cc:2454] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d82b51a88d87458f8b328bade8c74a69 in term 1.
I20260812 18:56:38.517802 983 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac136d01c64b49309be251f25b019fee, d82b51a88d87458f8b328bade8c74a69; no voters:
I20260812 18:56:38.518035 1282 raft_consensus.cc:2781] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:38.518306 1282 raft_consensus.cc:686] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 1 LEADER]: Becoming Leader. State: Replica: d82b51a88d87458f8b328bade8c74a69, State: Running, Role: LEADER
I20260812 18:56:38.518723 1282 consensus_queue.cc:237] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:38.522603 731 catalog_manager.cc:5495] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 reported cstate change: term changed from 0 to 1, leader changed from <none> to d82b51a88d87458f8b328bade8c74a69 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "d82b51a88d87458f8b328bade8c74a69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } health_report { overall_health: UNKNOWN } } }
I20260812 18:56:38.549353 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.545153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.16.40.194:35263)
0812 18:56:38.545293 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:38.545296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:38.545311 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:56:38.545459 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:38.545461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:38.545525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:38.545672 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:56:38.546021 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.546665 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.547638 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:38.547871 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:38.547999 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:38.548357 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:56:38.548899 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:56:38.549047 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:38.549094 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 18:56:38.549169 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 18:56:38.556061 1212 raft_consensus.cc:1261] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Refusing update from remote peer d82b51a88d87458f8b328bade8c74a69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:38.556057 899 raft_consensus.cc:1261] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Refusing update from remote peer d82b51a88d87458f8b328bade8c74a69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:38.556885 1282 consensus_queue.cc:1035] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:38.557166 1285 consensus_queue.cc:1035] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:38.564291 1289 mvcc.cc:205] Tried to move back new op lower bound from 7317753850077241344 to 7317753849935732736. Current Snapshot: MvccSnapshot[applied={T|T < 7317753850077241344}]
I20260812 18:56:38.567327 1290 mvcc.cc:205] Tried to move back new op lower bound from 7317753850077241344 to 7317753849935732736. Current Snapshot: MvccSnapshot[applied={T|T < 7317753850077241344}]
I20260812 18:56:38.574020 730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:38.574244 730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:38.577102 730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Sending DeleteTablet for 3 replicas of tablet 1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:38.578142 1039 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:38 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:34702
I20260812 18:56:38.578145 1188 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:38 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:41574
I20260812 18:56:38.578399 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ff63627285b49938c4eaa9d6f8541c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:38 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:53114
I20260812 18:56:38.578785 1302 tablet_replica.cc:330] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: stopping tablet replica
I20260812 18:56:38.578899 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 777
I20260812 18:56:38.578943 1302 raft_consensus.cc:2229] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:38.586094 1299 tablet_replica.cc:330] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: stopping tablet replica
I20260812 18:56:38.586494 1300 tablet_replica.cc:330] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: stopping tablet replica
I20260812 18:56:38.586670 1299 raft_consensus.cc:2229] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:56:38.587030 1300 raft_consensus.cc:2229] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:38.587212 1299 raft_consensus.cc:2258] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:38.587798 1300 raft_consensus.cc:2258] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:38.589280 1299 ts_tablet_manager.cc:1905] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:38.589919 1300 ts_tablet_manager.cc:1905] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:38.593228 1300 ts_tablet_manager.cc:1918] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:38.593500 1300 log.cc:1196] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:38.593637 1299 ts_tablet_manager.cc:1918] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:38.593865 1299 log.cc:1196] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:38.594338 1299 ts_tablet_manager.cc:1939] T 1ff63627285b49938c4eaa9d6f8541c0 P d82b51a88d87458f8b328bade8c74a69: Deleting consensus metadata
I20260812 18:56:38.594532 1300 ts_tablet_manager.cc:1939] T 1ff63627285b49938c4eaa9d6f8541c0 P 66bd3d3b8749483e9a5f0e5c0f6b0a41: Deleting consensus metadata
W20260812 18:56:38.595150 713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:44485 (error 108)
I20260812 18:56:38.596231 713 catalog_manager.cc:4841] TS d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): tablet 1ff63627285b49938c4eaa9d6f8541c0 (table test-table [id=c017fb5814904b879822d9872859cfbc]) successfully deleted
I20260812 18:56:38.596519 714 catalog_manager.cc:4841] TS 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): tablet 1ff63627285b49938c4eaa9d6f8541c0 (table test-table [id=c017fb5814904b879822d9872859cfbc]) successfully deleted
I20260812 18:56:38.597330 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 956
I20260812 18:56:38.597625 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:38.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44485 (local address 127.0.0.1:46610)
0812 18:56:38.597408 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:38.597510 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 18:56:38.608349 713 catalog_manager.cc:4584] TS ac136d01c64b49309be251f25b019fee (127.16.40.193:44485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ff63627285b49938c4eaa9d6f8541c0: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44485: connect: Connection refused (error 111)
I20260812 18:56:38.614374 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 1118
I20260812 18:56:38.627257 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 689
I20260812 18:56:38.639889 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--webserver_interface=127.16.40.254
--webserver_port=41171
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:38.762645 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:38.763022 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:38.763098 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:38.767354 1305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:38.767493 1305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:38.767549 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:38.767612 1305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:38.767650 1305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:38.773363 1305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:38.774621 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:38.775754 1305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:38.788216 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:38.788216 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:38.788257 1314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:38.788738 1305 server_base.cc:1034] running on GCE node
I20260812 18:56:38.789587 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:38.790971 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:38.792197 1305 hybrid_clock.cc:648] HybridClock initialized: now 1786560998792183 us; error 49 us; skew 500 ppm
Aug 12 18:56:38 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560998, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:38.795796 1305 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:38.797089 1305 webserver.cc:460] Webserver started at http://127.16.40.254:41171/ using document root <none> and password file <none>
I20260812 18:56:38.797775 1305 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:38.797884 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:38.802441 1305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:38.805269 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:38.806838 1305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:38.807049 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a706725a45ba4b5982bba101691058a9"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "4ecb5b808400825a15ea261e84f2320b"
server_key_iv: "051ced87b4796ed828677b20f9380479"
server_key_version: "encryptionkey@0"
I20260812 18:56:38.807716 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:38.827297 1305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:38.831097 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:38.831341 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:38.839466 1305 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38533
I20260812 18:56:38.839780 1382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38533 every 8 connection(s)
I20260812 18:56:38.840814 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:38.845557 1383 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:56:38.847635 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap starting.
I20260812 18:56:38.849768 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1305
I20260812 18:56:38.850252 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:44485
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=42687
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:38.871424 1383 log.cc:824] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:38.887568 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:56:38.888127 1383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9: Bootstrap complete.
I20260812 18:56:38.893936 1383 raft_consensus.cc:348] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:38.894493 1383 raft_consensus.cc:729] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a706725a45ba4b5982bba101691058a9, State: Initialized, Role: FOLLOWER
I20260812 18:56:38.895097 1383 consensus_queue.cc:260] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:38.895262 1383 raft_consensus.cc:388] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:38.895327 1383 raft_consensus.cc:482] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:38.895439 1383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 2 FOLLOWER]: Advancing to term 3
I20260812 18:56:38.898418 1383 raft_consensus.cc:504] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:38.898854 1383 leader_election.cc:302] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a706725a45ba4b5982bba101691058a9; no voters:
I20260812 18:56:38.899248 1383 leader_election.cc:288] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 18:56:38.899420 1388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 18:56:38.899977 1388 raft_consensus.cc:686] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [term 3 LEADER]: Becoming Leader. State: Replica: a706725a45ba4b5982bba101691058a9, State: Running, Role: LEADER
I20260812 18:56:38.900326 1388 consensus_queue.cc:237] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } }
I20260812 18:56:38.900718 1383 sys_catalog.cc:564] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:38.901844 1389 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:38.901983 1389 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:38.902326 1390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a706725a45ba4b5982bba101691058a9. Latest consensus state: current_term: 3 leader_uuid: "a706725a45ba4b5982bba101691058a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a706725a45ba4b5982bba101691058a9" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38533 } } }
I20260812 18:56:38.902426 1390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a706725a45ba4b5982bba101691058a9 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:38.903333 1397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:38.908226 1397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:38.908977 1397 catalog_manager.cc:1259] Loaded cluster ID: a9c809eec734445d81eef3020943925d
I20260812 18:56:38.909066 1397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 18:56:38.910073 1397 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8d4eb1d9ad google::LogMessage::Fail() at ??:0
@ 0x7f8d4eb2088a google::LogMessage::SendToLog() at ??:0
@ 0x7f8d4eb1d4de google::LogMessage::Flush() at ??:0
@ 0x7f8d4eb1ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8d5945cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8d5945dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8d5945a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8d5949e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8d50d6f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8d50d6fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8d50d713f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8d50d60512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8d523a46db start_thread at ??:0
@ 0x7f8d4d61b61f clone at ??:0
W20260812 18:56:39.029418 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:39.029911 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:39.030006 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:39.036612 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:39.036715 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:39.036810 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:39.045878 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:44485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=42687
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:39.047536 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:39.049077 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:56:39.063287 1385 server_base.cc:1034] running on GCE node
W20260812 18:56:39.063203 1426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:39.063201 1430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:39.063505 1428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:39.064051 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:39.065062 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:39.066293 1385 hybrid_clock.cc:648] HybridClock initialized: now 1786560999066247 us; error 57 us; skew 500 ppm
Aug 12 18:56:39 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560999, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:39.070029 1385 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:39.071095 1385 webserver.cc:460] Webserver started at http://127.16.40.193:42687/ using document root <none> and password file <none>
I20260812 18:56:39.071759 1385 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:39.071854 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:39.076628 1385 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:56:39.080120 1437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.081828 1385 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 18:56:39.082036 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "ac136d01c64b49309be251f25b019fee"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "d0f4795ffb270ca96746c220d24693a8"
server_key_iv: "928039ae984065b1094e480e57382542"
server_key_version: "encryptionkey@0"
I20260812 18:56:39.082772 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:39.098405 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:39.102341 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:39.102593 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:39.103284 1385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:39.104660 1445 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 18:56:39.109515 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 18:56:39.109637 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 18:56:39.109723 1385 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 18:56:39.112530 1385 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 18:56:39.112640 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:39.113099 1445 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Bootstrap starting.
I20260812 18:56:39.124401 1385 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:44485
I20260812 18:56:39.124432 1576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:44485 every 8 connection(s)
I20260812 18:56:39.125854 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:39.130851 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1385
I20260812 18:56:39.131371 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:35263
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=46257
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:56:39.136044 1445 log.cc:824] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:39.147631 1445 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:56:39.148324 1445 tablet_bootstrap.cc:492] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Bootstrap complete.
I20260812 18:56:39.150094 1445 ts_tablet_manager.cc:1397] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20260812 18:56:39.154043 1445 raft_consensus.cc:348] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:39.154840 1445 raft_consensus.cc:729] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac136d01c64b49309be251f25b019fee, State: Initialized, Role: FOLLOWER
I20260812 18:56:39.155543 1445 consensus_queue.cc:260] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:39.156857 1445 ts_tablet_manager.cc:1428] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260812 18:56:39.168648 1580 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:56:39.128523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:35803)
0812 18:56:39.129301 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.129368 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:39.130724 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:39.167704 (+ 36980us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":196,"threads_started":1}
W20260812 18:56:39.169329 1577 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:38533 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:38533: Client connection negotiation failed: client connection to 127.16.40.254:38533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 18:56:39.170192 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.169867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:36767)
0812 18:56:39.170042 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.170127 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 18:56:39.171021 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.170750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:52797)
0812 18:56:39.170889 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.170953 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 18:56:39.171264 1577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:56:39.265196 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:39.265529 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:39.265578 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:39.269953 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:39.270042 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:39.270164 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:39.276237 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:35263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=46257
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:39.277628 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:39.278833 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:39.290153 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:39.290153 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:39.290170 1581 server_base.cc:1034] running on GCE node
W20260812 18:56:39.290194 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:39.291023 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:39.291810 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:39.293040 1581 hybrid_clock.cc:648] HybridClock initialized: now 1786560999293022 us; error 33 us; skew 500 ppm
Aug 12 18:56:39 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560999, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:39.296061 1581 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:39.297160 1581 webserver.cc:460] Webserver started at http://127.16.40.194:46257/ using document root <none> and password file <none>
I20260812 18:56:39.297804 1581 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:39.297904 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:39.302238 1581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:56:39.304641 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.306049 1581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:39.306232 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "d82b51a88d87458f8b328bade8c74a69"
format_stamp: "Formatted at 2026-08-12 18:56:36 on dist-test-slave-rnrw"
server_key: "26a470f4f654674418ce4c103162bdc5"
server_key_iv: "5d50462c0b3fd61e41b8193b2069ee8b"
server_key_version: "encryptionkey@0"
I20260812 18:56:39.306835 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:39.337939 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:39.341454 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:39.341784 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:39.342450 1581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:39.343374 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:39.343448 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.343525 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:39.343577 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.353230 1581 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35263
I20260812 18:56:39.353253 1735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35263 every 8 connection(s)
I20260812 18:56:39.354406 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:39.359193 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.356903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:53947)
0812 18:56:39.357460 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.357592 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
W20260812 18:56:39.359929 1736 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:38533 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:38533: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
I20260812 18:56:39.360769 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.360451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:42773)
0812 18:56:39.360650 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.360703 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 18:56:39.361569 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.361264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:50047)
0812 18:56:39.361395 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.361513 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 18:56:39.361907 1736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 18:56:39.363077 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1581
I20260812 18:56:39.363566 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:39845
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:38533
--webserver_port=43509
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:39.484488 1739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:39.484858 1739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:39.484907 1739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:39.489905 1739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:39.490015 1739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:39.490090 1739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:39.496135 1739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=43509
--tserver_master_addrs=127.16.40.254:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:39.497442 1739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:39.498581 1739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:39.509943 1748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:39.509913 1745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:39.510211 1739 server_base.cc:1034] running on GCE node
W20260812 18:56:39.510375 1746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:39.510890 1739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:39.511693 1739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:39.512858 1739 hybrid_clock.cc:648] HybridClock initialized: now 1786560999512846 us; error 47 us; skew 500 ppm
Aug 12 18:56:39 dist-test-slave-rnrw krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560999, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:39.516079 1739 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:39.517310 1739 webserver.cc:460] Webserver started at http://127.16.40.195:43509/ using document root <none> and password file <none>
I20260812 18:56:39.517973 1739 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:39.518045 1739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:39.522534 1739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 18:56:39.525166 1755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.527206 1739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:56:39.527344 1739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41"
format_stamp: "Formatted at 2026-08-12 18:56:37 on dist-test-slave-rnrw"
server_key: "141c4027794b0cc16581ce72d37cb121"
server_key_iv: "d07b0cbaa0a4bcd922ca4df170b2baf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:39.527987 1739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:39.554467 1739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:39.557977 1739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:39.558156 1739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:39.558761 1739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:39.559623 1739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:39.559675 1739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.559718 1739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:39.559734 1739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:39.569731 1739 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39845
I20260812 18:56:39.569849 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39845 every 8 connection(s)
I20260812 18:56:39.570878 1739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:39.571985 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1739
I20260812 18:56:39.575711 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.573691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:40705)
0812 18:56:39.574227 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.574358 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":137,"threads_started":1}
W20260812 18:56:39.576303 1884 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:38533 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:38533: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
I20260812 18:56:39.577044 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.576776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:41573)
0812 18:56:39.576918 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.576972 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 18:56:39.577723 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:39.577490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:43323)
0812 18:56:39.577623 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:39.577670 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 18:56:39.578068 1884 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 18:56:40.172710 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.172019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:48813)
0812 18:56:40.172371 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:40.172464 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260812 18:56:40.363107 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.362609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:47519)
0812 18:56:40.362890 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:40.362978 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260812 18:56:40.579282 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.578746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:52121)
0812 18:56:40.579047 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:40.579144 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260812 18:56:40.760160 1891 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:40.760524 1891 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:40.761893 1891 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
Aug 12 18:56:40 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560999, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.195@KRBTEST.COM
Aug 12 18:56:40 dist-test-slave-rnrw krb5kdc[32528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786560999, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.194@KRBTEST.COM
I20260812 18:56:40.778499 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.762473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:48761 (local address 127.16.40.194:35263)
0812 18:56:40.762975 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:40.763001 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:40.763046 (+ 45us) server_negotiation.cc:407] Connection header received
0812 18:56:40.763774 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:40.763781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:40.764127 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:40.764446 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:40.765854 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.766844 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768201 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.768767 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768971 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:40.772308 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:40.772314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:40.772327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:40.772361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:40.774719 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:40.775722 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:40.775729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:40.775736 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:40.775928 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:40.776766 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:40.776769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:40.776770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:40.777194 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:40.777359 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:40.778021 (+ 662us) server_negotiation.cc:299] Negotiation successful
0812 18:56:40.778300 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":114,"threads_started":1}
I20260812 18:56:40.778486 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.762142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35263 (local address 127.16.40.193:48761)
0812 18:56:40.762534 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:40.762551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:40.762706 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:40.764897 (+ 2191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:40.764899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:40.765142 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:40.765654 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:40.765672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.767016 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.767020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:40.767998 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:40.768009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.768915 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:40.769206 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:40.769937 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:40.769955 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:40.772084 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:40.774897 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:40.774906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:40.774925 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:40.775540 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:40.776505 (+ 965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:40.776508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:40.776510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:40.776641 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:40.777325 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:40.777332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:40.777656 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 18:56:40.777887 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 18:56:40.778287 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":162,"spinlock_wait_cycles":53248,"thread_start_us":100,"threads_started":1}
I20260812 18:56:40.778903 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.762466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:40279 (local address 127.16.40.195:39845)
0812 18:56:40.762933 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:40.762963 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:40.763037 (+ 74us) server_negotiation.cc:407] Connection header received
0812 18:56:40.763749 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:40.763755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:40.764134 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:40.764426 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:40.765872 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.766845 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768183 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.768768 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768984 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:40.772140 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:40.772150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:40.772173 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:40.772219 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:40.774702 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:40.775722 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:40.775729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:40.775735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:40.775928 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:40.776480 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:40.776486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:40.776490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:40.777238 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:40.777390 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:40.778347 (+ 957us) server_negotiation.cc:299] Negotiation successful
0812 18:56:40.778657 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":95,"threads_started":1}
I20260812 18:56:40.779439 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:40.762151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39845 (local address 127.16.40.193:40279)
0812 18:56:40.762534 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:40.762551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:40.762671 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:40.764595 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:40.764611 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:40.765155 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:40.765655 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:40.765673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.767023 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.767026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:40.767999 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:40.768009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:40.768940 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:40.768944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:40.769205 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:40.769991 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:40.770004 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:40.771803 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:40.774924 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:40.774927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:40.774929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:40.775540 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:40.776108 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:40.776111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:40.776113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:40.776348 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:40.777450 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:40.777455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:40.777659 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 18:56:40.777887 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 18:56:40.779185 (+ 1298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":1093,"spinlock_wait_cycles":78336,"thread_start_us":67,"threads_started":1}
W20260812 18:56:40.783147 1440 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
W20260812 18:56:40.783385 1441 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:40.783475 1441 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac136d01c64b49309be251f25b019fee; no voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, d82b51a88d87458f8b328bade8c74a69
I20260812 18:56:40.783805 1891 raft_consensus.cc:2726] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:56:41.174341 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:41.173797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:52173)
0812 18:56:41.174102 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:41.174187 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260812 18:56:41.364704 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:41.364147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:45721)
0812 18:56:41.364454 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:41.364523 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260812 18:56:41.580770 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:41.580202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:51333)
0812 18:56:41.580500 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:41.580626 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
I20260812 18:56:42.175832 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:42.175345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:32991)
0812 18:56:42.175667 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:42.175728 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260812 18:56:42.305337 1900 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:42.305482 1900 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:42.305930 1900 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
W20260812 18:56:42.306844 1440 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
W20260812 18:56:42.307041 1441 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:42.307152 1441 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac136d01c64b49309be251f25b019fee; no voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, d82b51a88d87458f8b328bade8c74a69
I20260812 18:56:42.307341 1900 raft_consensus.cc:2726] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:56:42.366369 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:42.365778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:53159)
0812 18:56:42.366147 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:42.366222 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260812 18:56:42.582233 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:42.581716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:37667)
0812 18:56:42.582045 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:42.582115 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
I20260812 18:56:43.177364 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:43.176887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:41985)
0812 18:56:43.177200 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:43.177265 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
I20260812 18:56:43.367751 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:43.367267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:47853)
0812 18:56:43.367581 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:43.367649 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260812 18:56:43.583834 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:43.583343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:58011)
0812 18:56:43.583657 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:43.583725 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260812 18:56:44.179191 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:44.178610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:53385)
0812 18:56:44.179007 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:44.179087 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":83,"threads_started":1}
I20260812 18:56:44.369287 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:44.368804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:42093)
0812 18:56:44.369114 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:44.369183 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260812 18:56:44.585346 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:44.584842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:44245)
0812 18:56:44.585173 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:44.585244 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
I20260812 18:56:45.180727 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:45.180215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:57797)
0812 18:56:45.180509 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:45.180618 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260812 18:56:45.370865 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:45.370322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:55355)
0812 18:56:45.370690 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:45.370761 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
I20260812 18:56:45.475618 1912 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:45.475761 1912 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:45.476305 1912 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
W20260812 18:56:45.477171 1440 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
W20260812 18:56:45.477278 1441 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:45.477394 1441 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac136d01c64b49309be251f25b019fee; no voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, d82b51a88d87458f8b328bade8c74a69
I20260812 18:56:45.477602 1912 raft_consensus.cc:2726] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:56:45.586669 1913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:45.586197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:57941)
0812 18:56:45.586473 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:45.586536 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260812 18:56:46.182132 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:46.181666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:46955)
0812 18:56:46.181960 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:46.182028 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260812 18:56:46.372417 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:46.371845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:32941)
0812 18:56:46.372248 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:46.372313 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":113,"threads_started":1}
I20260812 18:56:46.588264 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:46.587771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:42289)
0812 18:56:46.588096 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:46.588163 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
I20260812 18:56:47.183724 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:47.183185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:52675)
0812 18:56:47.183491 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:47.183565 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":118,"threads_started":1}
I20260812 18:56:47.374045 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:47.373543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:48643)
0812 18:56:47.373859 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:47.373936 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260812 18:56:47.589558 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:47.589102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:55803)
0812 18:56:47.589399 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:47.589461 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260812 18:56:48.185271 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:48.184714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:42637)
0812 18:56:48.185081 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:48.185157 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":83,"threads_started":1}
I20260812 18:56:48.375597 1922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:48.375134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:39115)
0812 18:56:48.375426 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:48.375494 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260812 18:56:48.591148 1923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:48.590585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:52053)
0812 18:56:48.590913 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:48.590983 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260812 18:56:49.186908 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:49.186286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:56065)
0812 18:56:49.186711 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:49.186805 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":105,"threads_started":1}
I20260812 18:56:49.377055 1925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:49.376632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:40603)
0812 18:56:49.376892 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:49.376954 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260812 18:56:49.531098 1926 raft_consensus.cc:482] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:49.531318 1926 raft_consensus.cc:504] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82b51a88d87458f8b328bade8c74a69" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35263 } } peers { permanent_uuid: "66bd3d3b8749483e9a5f0e5c0f6b0a41" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39845 } } peers { permanent_uuid: "ac136d01c64b49309be251f25b019fee" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44485 } }
I20260812 18:56:49.531934 1926 leader_election.cc:288] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263), 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845)
W20260812 18:56:49.532919 1441 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 66bd3d3b8749483e9a5f0e5c0f6b0a41 (127.16.40.195:39845): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
W20260812 18:56:49.533020 1440 leader_election.cc:341] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d82b51a88d87458f8b328bade8c74a69 (127.16.40.194:35263): Not found: Tablet not found: 1ff63627285b49938c4eaa9d6f8541c0
I20260812 18:56:49.533066 1441 leader_election.cc:302] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac136d01c64b49309be251f25b019fee; no voters: 66bd3d3b8749483e9a5f0e5c0f6b0a41, d82b51a88d87458f8b328bade8c74a69
I20260812 18:56:49.533233 1926 raft_consensus.cc:2726] T 1ff63627285b49938c4eaa9d6f8541c0 P ac136d01c64b49309be251f25b019fee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:56:49.592509 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:49.592046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:48899)
0812 18:56:49.592340 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:49.592407 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260812 18:56:50.188336 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:50.187823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:46577)
0812 18:56:50.188160 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:50.188230 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
I20260812 18:56:50.378479 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:50.377992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:40671)
0812 18:56:50.378308 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:50.378375 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260812 18:56:50.593986 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:50.593504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:45719)
0812 18:56:50.593797 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:50.593880 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":83,"threads_started":1}
I20260812 18:56:51.189867 1931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:51.189352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:43821)
0812 18:56:51.189690 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:51.189758 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260812 18:56:51.380105 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:51.379567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:34567)
0812 18:56:51.379912 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:51.379995 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260812 18:56:51.595717 1934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:51.595112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:46297)
0812 18:56:51.595496 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:51.595580 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260812 18:56:52.191227 1935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:52.190792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:53475)
0812 18:56:52.191063 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:52.191121 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260812 18:56:52.381601 1936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:52.381115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:37045)
0812 18:56:52.381436 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:52.381492 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
I20260812 18:56:52.597280 1937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:52.596768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:37683)
0812 18:56:52.597106 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:52.597178 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":80,"threads_started":1}
I20260812 18:56:53.192611 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:53.192098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:58443)
0812 18:56:53.192418 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:53.192488 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":87,"threads_started":1}
I20260812 18:56:53.383179 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:53.382586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:57203)
0812 18:56:53.382996 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:53.383071 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":129,"threads_started":1}
I20260812 18:56:53.598838 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:53.598328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:46515)
0812 18:56:53.598664 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:53.598733 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260812 18:56:54.194351 1941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:54.193819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.193:48399)
0812 18:56:54.194173 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:54.194246 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260812 18:56:54.384928 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:54.384338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.194:40997)
0812 18:56:54.384743 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:54.384815 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":94,"threads_started":1}
I20260812 18:56:54.579602 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 1385
I20260812 18:56:54.591605 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 1581
I20260812 18:56:54.600665 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:54.600046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:38533 (local address 127.16.40.195:36973)
0812 18:56:54.600359 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:54.600432 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
I20260812 18:56:54.601289 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 1739
2026-08-12T18:56:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19669 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:56:54 dist-test-slave-rnrw krb5kdc[1947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:56:54 dist-test-slave-rnrw krb5kdc[1947](info): set up 2 sockets
Aug 12 18:56:54 dist-test-slave-rnrw krb5kdc[1947](info): commencing operation
krb5kdc: starting...
W20260812 18:56:55.789254 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:56:55 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:56:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:56:55.937309 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:56.067869 1963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:56.068204 1963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:56.068248 1963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:56.072678 1963 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 18:56:56.072763 1963 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 18:56:56.072779 1963 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 18:56:56.072794 1963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:56:56.072813 1963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:56.072832 1963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:56.072880 1963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:56:56.072918 1963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:56:56.079021 1963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:56.080354 1963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:56.081527 1963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:56.092413 1969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.092638 1963 server_base.cc:1034] running on GCE node
W20260812 18:56:56.092422 1972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.092442 1968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.093712 1963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:56.095100 1963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:56.096412 1963 hybrid_clock.cc:648] HybridClock initialized: now 1786561016096366 us; error 80 us; skew 500 ppm
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:56.099879 1963 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:56:56.101109 1963 webserver.cc:460] Webserver started at http://127.16.40.254:40669/ using document root <none> and password file <none>
I20260812 18:56:56.101743 1963 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:56.101814 1963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:56.102002 1963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:56.104354 1963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "a7cb6254fd15439bbe7bd197bed76448"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "68eda0b4f8fa679a85795d594fe03561"
server_key_iv: "066e2494ce7e3e6c812fe23ac0975005"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.105054 1963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "a7cb6254fd15439bbe7bd197bed76448"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "68eda0b4f8fa679a85795d594fe03561"
server_key_iv: "066e2494ce7e3e6c812fe23ac0975005"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.109534 1963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:56:56.112635 1981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.114192 1963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:56.114368 1963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a7cb6254fd15439bbe7bd197bed76448"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "68eda0b4f8fa679a85795d594fe03561"
server_key_iv: "066e2494ce7e3e6c812fe23ac0975005"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.114583 1963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:56.127070 1963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:56.131264 1963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:56.131487 1963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:56.139408 1963 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:40205
I20260812 18:56:56.139746 2041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:40205 every 8 connection(s)
I20260812 18:56:56.140523 1963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:56:56.143774 2042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:56.146508 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 1963
I20260812 18:56:56.146662 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:56:56.146975 16547 external_mini_cluster.cc:1428] Setting key 42c78a9ed2d04db0af53777365ca1f4b
I20260812 18:56:56.151245 2042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap starting.
I20260812 18:56:56.154148 2042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:56.155184 2042 log.cc:824] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:56.157977 2042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: No bootstrap required, opened a new log
I20260812 18:56:56.162139 2042 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:56:56.162451 2042 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:56.162549 2042 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Initialized, Role: FOLLOWER
I20260812 18:56:56.163053 2042 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:56:56.163225 2042 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:56:56.163319 2042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:56:56.163457 2042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:56.164054 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.148738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.16.40.254:40205)
0812 18:56:56.149240 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:56.149259 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:56.149301 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:56:56.150124 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:56.150144 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:56.150545 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:56.150869 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:56.151865 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.153059 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.154318 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.154864 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.155015 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.158277 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:56.158284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:56.158300 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:56.158332 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:56.160277 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.160909 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.160915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.160921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.161060 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.161381 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.161385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.161388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.161878 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:56.162131 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:56.162461 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 18:56:56.163013 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":163,"threads_started":1}
I20260812 18:56:56.164703 2042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:56:56.165082 2042 leader_election.cc:302] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7cb6254fd15439bbe7bd197bed76448; no voters:
I20260812 18:56:56.165493 2042 leader_election.cc:288] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:56:56.165596 2048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:56.165879 2048 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 LEADER]: Becoming Leader. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Running, Role: LEADER
I20260812 18:56:56.166297 2048 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:56:56.166939 2042 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:56:56.167655 2048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cb6254fd15439bbe7bd197bed76448. Latest consensus state: current_term: 1 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:56:56.167898 2048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:56.168114 2049 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:56:56.168247 2049 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:56:56.168466 2056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:56:56.172124 2056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:56:56.179013 2056 catalog_manager.cc:1347] Generated new cluster ID: 88b7c5e4848549e59cb86815cca41fdb
I20260812 18:56:56.179562 2056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:56:56.192945 2056 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:56:56.194082 2056 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:56:56.201773 2056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Generated new TSK 0
I20260812 18:56:56.202788 2056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:56:56.244179 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:56.376262 2070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:56.376745 2070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:56.376852 2070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:56.382148 2070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:56.382292 2070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:56.382400 2070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:56:56.388631 2070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:56.389957 2070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:56.391258 2070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:56.402541 2077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.402712 2070 server_base.cc:1034] running on GCE node
W20260812 18:56:56.402946 2079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.402994 2076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.403652 2070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:56.404513 2070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:56.405834 2070 hybrid_clock.cc:648] HybridClock initialized: now 1786561016405811 us; error 41 us; skew 500 ppm
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:56.413625 2070 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:56:56.414844 2070 webserver.cc:460] Webserver started at http://127.16.40.193:32885/ using document root <none> and password file <none>
I20260812 18:56:56.415627 2070 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:56.415756 2070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:56.416044 2070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:56.418625 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "4f20b5b385f946718d32186057ad5913"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "cf9874ef9ac00e4057e7a02b9ef81353"
server_key_iv: "c360b538b88228549974089e4eda6bf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.419431 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "4f20b5b385f946718d32186057ad5913"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "cf9874ef9ac00e4057e7a02b9ef81353"
server_key_iv: "c360b538b88228549974089e4eda6bf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.424252 2070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 18:56:56.427714 2087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.429508 2070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:56.429695 2070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "4f20b5b385f946718d32186057ad5913"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "cf9874ef9ac00e4057e7a02b9ef81353"
server_key_iv: "c360b538b88228549974089e4eda6bf3"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.429840 2070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:56.439710 2070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:56.443537 2070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:56.443820 2070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:56.444617 2070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:56.445598 2070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:56.445653 2070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.445698 2070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:56.445714 2070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.456348 2070 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:34591
I20260812 18:56:56.456395 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:34591 every 8 connection(s)
I20260812 18:56:56.457420 2070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:56:56.463848 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2070
I20260812 18:56:56.463965 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:56:56.464298 16547 external_mini_cluster.cc:1428] Setting key e5b25ec5b0ea246a7dcd8a01b4d23979
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:56:56.475417 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.459618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:58943 (local address 127.16.40.254:40205)
0812 18:56:56.459842 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:56.459846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:56.460824 (+ 978us) server_negotiation.cc:407] Connection header received
0812 18:56:56.462466 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:56.462471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:56.462545 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:56.462637 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:56.464237 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.465163 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.466777 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.467129 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.467276 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.471020 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:56.471027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:56.471033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:56.471089 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:56.472966 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.473649 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.473652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.473653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.473740 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.474145 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.474149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.474151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.474367 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:56.474501 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:56.475086 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 18:56:56.475240 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 18:56:56.476454 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.459960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:58943)
0812 18:56:56.460639 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:56.460694 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:56.462172 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:56.462860 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:56.462869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:56.463372 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:56.464046 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.464067 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.465337 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.465340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.466613 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.466628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.467364 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.467370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.467668 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.468471 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:56.468492 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:56.470759 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:56.473135 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.473143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.473166 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.473506 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.473844 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.473847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.473849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.474036 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.474494 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:56.474502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:56.474820 (+ 318us) client_negotiation.cc:769] Sending connection context
0812 18:56:56.475046 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 18:56:56.475452 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":134,"threads_started":1}
I20260812 18:56:56.478178 2218 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:56:56.478619 2218 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:56:56.479283 2218 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:56:56.481350 2003 ts_manager.cc:194] Registered new tserver with Master: 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:56:56.482888 2003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:58943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:56:56.501125 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:56.628512 2223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:56.628930 2223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:56.628979 2223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:56.633399 2223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:56.633520 2223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:56.633590 2223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:56:56.639483 2223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:56.640755 2223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:56.641849 2223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:56.652590 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.652618 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.652602 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.652956 2223 server_base.cc:1034] running on GCE node
I20260812 18:56:56.653494 2223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:56.654251 2223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:56.655546 2223 hybrid_clock.cc:648] HybridClock initialized: now 1786561016655509 us; error 47 us; skew 500 ppm
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:56.658730 2223 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:56:56.659837 2223 webserver.cc:460] Webserver started at http://127.16.40.194:37281/ using document root <none> and password file <none>
I20260812 18:56:56.660477 2223 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:56.660610 2223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:56.660879 2223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:56.663189 2223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "7ef4cb5d75f93066e74262909bb8f57c"
server_key_iv: "21552df2342f9c5afd494106a76baa0f"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.663821 2223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "7ef4cb5d75f93066e74262909bb8f57c"
server_key_iv: "21552df2342f9c5afd494106a76baa0f"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.668634 2223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 18:56:56.671607 2239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.673192 2223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:56.673341 2223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "7ef4cb5d75f93066e74262909bb8f57c"
server_key_iv: "21552df2342f9c5afd494106a76baa0f"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.673496 2223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:56.689558 2223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:56.693702 2223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:56.693949 2223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:56.694676 2223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:56.695685 2223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:56.695740 2223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.695818 2223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:56.695856 2223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.706674 2223 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33735
I20260812 18:56:56.706729 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33735 every 8 connection(s)
I20260812 18:56:56.707769 2223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:56:56.710391 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2223
I20260812 18:56:56.710500 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:56:56.710795 16547 external_mini_cluster.cc:1428] Setting key 54dee1775fd31a4ccd6848bab192df56
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:56:56.724319 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.710108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:39873 (local address 127.16.40.254:40205)
0812 18:56:56.710326 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:56.710331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:56.711616 (+ 1285us) server_negotiation.cc:407] Connection header received
0812 18:56:56.712966 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:56.712972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:56.713042 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:56.713143 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:56.714670 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.715224 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.716502 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.716813 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.716942 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.719927 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:56.719931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:56.719934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:56.719973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:56.721732 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.722414 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.722418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.722420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.722524 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.722950 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.722953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.722955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.723251 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:56.723380 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:56.723972 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 18:56:56.724140 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:56:56.725471 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.710625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:39873)
0812 18:56:56.711192 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:56.711258 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:56.712721 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:56.713379 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:56.713387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:56.713882 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:56.714470 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.714494 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.715390 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.715393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.716348 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.716360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.717007 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.717011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.717220 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.717930 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:56.717949 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:56.719681 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:56.721893 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.721901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.721927 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.722285 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.722649 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.722652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.722654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.722837 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.723381 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:56.723390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:56.723704 (+ 314us) client_negotiation.cc:769] Sending connection context
0812 18:56:56.723926 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 18:56:56.724325 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":162,"threads_started":1}
I20260812 18:56:56.726969 2373 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:56:56.727308 2373 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:56.728055 2373 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:56:56.729622 2003 ts_manager.cc:194] Registered new tserver with Master: f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735)
I20260812 18:56:56.730278 2003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:39873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:56:56.746245 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:56:56.870970 2378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:56:56.871286 2378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:56:56.871332 2378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:56:56.875782 2378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:56:56.875865 2378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:56:56.875960 2378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:56:56.881981 2378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:56:56.883209 2378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:56:56.884402 2378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:56:56.895265 2387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.895265 2384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:56:56.895265 2383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:56:56.895284 2378 server_base.cc:1034] running on GCE node
I20260812 18:56:56.896134 2378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:56:56.897038 2378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:56:56.898268 2378 hybrid_clock.cc:648] HybridClock initialized: now 1786561016898240 us; error 48 us; skew 500 ppm
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:56:56.901882 2378 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:56:56.903009 2378 webserver.cc:460] Webserver started at http://127.16.40.195:45049/ using document root <none> and password file <none>
I20260812 18:56:56.903692 2378 fs_manager.cc:359] Metadata directory not provided
I20260812 18:56:56.903766 2378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:56:56.903962 2378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:56:56.906529 2378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "9b25ab2635074f14b03cfed6e359d403"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "2e3b6c50930ef1716f776cadf59637b4"
server_key_iv: "239008b0b8ed8507714bb5a3bfc5e530"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.907217 2378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "9b25ab2635074f14b03cfed6e359d403"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "2e3b6c50930ef1716f776cadf59637b4"
server_key_iv: "239008b0b8ed8507714bb5a3bfc5e530"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.912096 2378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 18:56:56.915501 2397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.917219 2378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:56:56.917387 2378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "9b25ab2635074f14b03cfed6e359d403"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "2e3b6c50930ef1716f776cadf59637b4"
server_key_iv: "239008b0b8ed8507714bb5a3bfc5e530"
server_key_version: "encryptionkey@0"
I20260812 18:56:56.917534 2378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:56:56.935456 2378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:56:56.939213 2378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:56:56.939450 2378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:56:56.940114 2378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:56:56.941234 2378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:56:56.941289 2378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.941363 2378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:56:56.941406 2378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:56:56.951062 2378 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:45421
I20260812 18:56:56.951120 2532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:45421 every 8 connection(s)
I20260812 18:56:56.952100 2378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:56:56.955145 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2378
I20260812 18:56:56.955389 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:56:56.955780 16547 external_mini_cluster.cc:1428] Setting key 0411467ab924db5b455d4687dfbc1d9e
Aug 12 18:56:56 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:56:56.968930 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.954313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:39321 (local address 127.16.40.254:40205)
0812 18:56:56.954488 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:56.954492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:56.955435 (+ 943us) server_negotiation.cc:407] Connection header received
0812 18:56:56.957304 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:56.957307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:56.957387 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:56.957548 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:56.959272 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.959862 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.961251 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.961480 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.961613 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.964703 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:56.964707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:56.964709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:56.964741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:56.966362 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.967011 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.967014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.967016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.967102 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.967570 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.967572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.967574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.967867 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:56.967951 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:56.968615 (+ 664us) server_negotiation.cc:299] Negotiation successful
0812 18:56:56.968754 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:56:56.970077 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.954703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:39321)
0812 18:56:56.955251 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:56.955312 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:56.957027 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:56.957857 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:56.957868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:56.958456 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:56:56.959114 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.959130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.960014 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.960017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.961098 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:56.961111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.961712 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.961714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:56.961879 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.962558 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:56:56.962576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:56:56.964347 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:56:56.966522 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.966528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.966577 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.966900 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.967192 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:56:56.967195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:56:56.967196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:56:56.967359 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:56:56.967965 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:56:56.967971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:56:56.968275 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 18:56:56.968460 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 18:56:56.968961 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260812 18:56:56.971338 2533 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:56:56.971608 2533 heartbeater.cc:464] Registering TS with master...
I20260812 18:56:56.972159 2533 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:56:56.973343 2003 ts_manager.cc:194] Registered new tserver with Master: 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
I20260812 18:56:56.973964 2003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:39321
I20260812 18:56:56.982744 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:56:56.992954 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:56.985033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.16.40.254:40205)
0812 18:56:56.985187 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:56.985191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:56.985313 (+ 122us) server_negotiation.cc:407] Connection header received
0812 18:56:56.985432 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:56.985434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:56.985481 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:56.985591 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:56:56.986289 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.986836 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.987797 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:56.988019 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:56.988109 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:56.989695 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:56:56.989698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:56:56.989700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:56:56.989727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:56:56.991121 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.991696 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.991699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.991700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.991779 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:56:56.992086 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:56:56.992089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:56:56.992090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:56:56.992320 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:56:56.992426 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:56.992716 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 18:56:56.992823 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:56:56.996426 2003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:56:56.999761 2003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:56:57.016242 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.010288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.16.40.193:34591)
0812 18:56:57.010669 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.010679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.011150 (+ 471us) server_negotiation.cc:407] Connection header received
0812 18:56:57.011387 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.011392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.011641 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.011777 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:57.012324 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.013373 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.014837 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.015422 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.015557 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.015736 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.015971 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.016078 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":79,"threads_started":1}
I20260812 18:56:57.016229 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.010088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.0.0.1:58114)
0812 18:56:57.010976 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.011040 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:57.011210 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:57.012006 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:57.012012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:57.012037 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:57.012183 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.012194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.013541 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.013545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.014701 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.014712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.015559 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.015562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.015674 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.015797 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 18:56:57.015926 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 18:56:57.016065 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":34,"thread_start_us":87,"threads_started":1}
I20260812 18:56:57.017365 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.011152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:45421 (local address 127.0.0.1:33180)
0812 18:56:57.011554 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.011566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:57.011642 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:57.012274 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:57.012277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:57.012297 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:57.012411 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.012421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.013924 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.013928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.015448 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.015461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.016827 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.016831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.016982 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.017037 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 18:56:57.017113 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 18:56:57.017218 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":212,"thread_start_us":120,"threads_started":1}
I20260812 18:56:57.017571 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.010174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.16.40.194:33735)
0812 18:56:57.010669 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.010680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.011341 (+ 661us) server_negotiation.cc:407] Connection header received
0812 18:56:57.011906 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.011916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.012121 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.012277 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:57.012743 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.013887 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.015622 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.016487 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.016611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.016755 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.017231 (+ 476us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.017337 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":147,"threads_started":1}
I20260812 18:56:57.017998 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.011499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.16.40.195:45421)
0812 18:56:57.011885 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.011894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.011912 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:56:57.011980 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.011983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.012128 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.012295 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:57.012580 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.013762 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.015616 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.016468 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.016611 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.016755 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.017789 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.017880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":179,"threads_started":1}
I20260812 18:56:57.018405 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.009978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33735 (local address 127.0.0.1:41704)
0812 18:56:57.010976 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.011040 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:57.011192 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:57.012462 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:57.012465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:57.012483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:57.012607 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.012615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.014032 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.014035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.015448 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.015465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.016827 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.016831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.017028 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.017045 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 18:56:57.017113 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 18:56:57.018188 (+ 1075us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":1104,"thread_start_us":108,"threads_started":1}
I20260812 18:56:57.020371 2143 tablet_service.cc:1467] Processing CreateTablet for tablet 203e484ab20e486e950a83cb331fff8e (DEFAULT_TABLE table=test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:57.021139 2455 tablet_service.cc:1467] Processing CreateTablet for tablet 203e484ab20e486e950a83cb331fff8e (DEFAULT_TABLE table=test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:57.021325 2295 tablet_service.cc:1467] Processing CreateTablet for tablet 203e484ab20e486e950a83cb331fff8e (DEFAULT_TABLE table=test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:56:57.021800 2143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e484ab20e486e950a83cb331fff8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:57.022195 2455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e484ab20e486e950a83cb331fff8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:57.022363 2295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203e484ab20e486e950a83cb331fff8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:56:57.029325 2556 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Bootstrap starting.
I20260812 18:56:57.030933 2558 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Bootstrap starting.
I20260812 18:56:57.032513 2556 tablet_bootstrap.cc:654] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:57.033017 2557 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Bootstrap starting.
I20260812 18:56:57.033401 2558 tablet_bootstrap.cc:654] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:57.034040 2556 log.cc:824] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:57.034253 2558 log.cc:824] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:57.035432 2557 tablet_bootstrap.cc:654] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Neither blocks nor log segments found. Creating new log.
I20260812 18:56:57.036180 2557 log.cc:824] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Log is configured to *not* fsync() on all Append() calls
I20260812 18:56:57.036574 2556 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: No bootstrap required, opened a new log
I20260812 18:56:57.036574 2558 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: No bootstrap required, opened a new log
I20260812 18:56:57.036795 2556 ts_tablet_manager.cc:1397] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 18:56:57.036795 2558 ts_tablet_manager.cc:1397] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:56:57.038560 2557 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: No bootstrap required, opened a new log
I20260812 18:56:57.038802 2557 ts_tablet_manager.cc:1397] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:56:57.040995 2558 raft_consensus.cc:348] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.040997 2556 raft_consensus.cc:348] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.041328 2558 raft_consensus.cc:374] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:57.041328 2556 raft_consensus.cc:374] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:57.041392 2556 raft_consensus.cc:729] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b25ab2635074f14b03cfed6e359d403, State: Initialized, Role: FOLLOWER
I20260812 18:56:57.041426 2558 raft_consensus.cc:729] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ba1ee9905e4553b54853fe95dda7a9, State: Initialized, Role: FOLLOWER
I20260812 18:56:57.041990 2556 consensus_queue.cc:260] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.041990 2558 consensus_queue.cc:260] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.042699 2557 raft_consensus.cc:348] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.043026 2533 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
I20260812 18:56:57.043051 2557 raft_consensus.cc:374] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:56:57.043150 2558 ts_tablet_manager.cc:1428] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:56:57.043150 2556 ts_tablet_manager.cc:1428] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:56:57.043195 2557 raft_consensus.cc:729] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f20b5b385f946718d32186057ad5913, State: Initialized, Role: FOLLOWER
I20260812 18:56:57.043654 2373 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
I20260812 18:56:57.043677 2557 consensus_queue.cc:260] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.044608 2218 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
I20260812 18:56:57.046308 2557 ts_tablet_manager.cc:1428] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:56:57.052244 2563 raft_consensus.cc:482] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:56:57.052480 2563 raft_consensus.cc:504] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.053874 2563 leader_election.cc:288] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:56:57.058153 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.054279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:58331 (local address 127.16.40.194:33735)
0812 18:56:57.054404 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.054410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.054819 (+ 409us) server_negotiation.cc:407] Connection header received
0812 18:56:57.054937 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.054946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.055000 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.055083 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:57.055363 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.055871 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057141 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.057571 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057705 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.057765 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.057953 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.058028 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 18:56:57.058183 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.054124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33735 (local address 127.16.40.195:58331)
0812 18:56:57.054724 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.054734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:57.054848 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:57.055094 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:57.055097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:57.055112 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:57.055245 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.055250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.055989 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.055994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.056986 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.056998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.057701 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.057803 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.057835 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 18:56:57.057926 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 18:56:57.058007 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":313,"thread_start_us":143,"threads_started":1}
I20260812 18:56:57.058346 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.054124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.16.40.195:34559)
0812 18:56:57.054269 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.054286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:56:57.054386 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:56:57.055160 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:56:57.055163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:56:57.055184 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:56:57.055305 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.055310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.056100 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.056108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.057067 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:56:57.057079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.057878 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:56:57.058003 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.058022 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 18:56:57.058121 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 18:56:57.058201 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 18:56:57.058346 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.054779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:34559 (local address 127.16.40.193:34591)
0812 18:56:57.054930 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.054934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.054948 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:56:57.054999 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.055002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.055043 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.055149 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:56:57.055433 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.055978 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057223 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.057727 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.057889 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.057944 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.058142 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.058221 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 18:56:57.058893 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e484ab20e486e950a83cb331fff8e" candidate_uuid: "9b25ab2635074f14b03cfed6e359d403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" is_pre_election: true
I20260812 18:56:57.059052 2166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e484ab20e486e950a83cb331fff8e" candidate_uuid: "9b25ab2635074f14b03cfed6e359d403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f20b5b385f946718d32186057ad5913" is_pre_election: true
I20260812 18:56:57.059306 2319 raft_consensus.cc:2454] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b25ab2635074f14b03cfed6e359d403 in term 0.
I20260812 18:56:57.059377 2166 raft_consensus.cc:2454] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b25ab2635074f14b03cfed6e359d403 in term 0.
I20260812 18:56:57.059911 2399 leader_election.cc:302] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f20b5b385f946718d32186057ad5913, 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9; no voters:
I20260812 18:56:57.060246 2563 raft_consensus.cc:2781] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:56:57.060359 2563 raft_consensus.cc:482] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:56:57.060415 2563 raft_consensus.cc:3037] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:57.061622 2563 raft_consensus.cc:504] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.062070 2563 leader_election.cc:288] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [CANDIDATE]: Term 1 election: Requested vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:56:57.062480 2319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e484ab20e486e950a83cb331fff8e" candidate_uuid: "9b25ab2635074f14b03cfed6e359d403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
I20260812 18:56:57.062521 2166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203e484ab20e486e950a83cb331fff8e" candidate_uuid: "9b25ab2635074f14b03cfed6e359d403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f20b5b385f946718d32186057ad5913"
I20260812 18:56:57.062635 2319 raft_consensus.cc:3037] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:57.062686 2166 raft_consensus.cc:3037] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:56:57.063848 2319 raft_consensus.cc:2454] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b25ab2635074f14b03cfed6e359d403 in term 1.
I20260812 18:56:57.063884 2166 raft_consensus.cc:2454] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b25ab2635074f14b03cfed6e359d403 in term 1.
I20260812 18:56:57.064291 2400 leader_election.cc:302] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9; no voters:
I20260812 18:56:57.064572 2563 raft_consensus.cc:2781] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:56:57.064919 2563 raft_consensus.cc:686] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 1 LEADER]: Becoming Leader. State: Replica: 9b25ab2635074f14b03cfed6e359d403, State: Running, Role: LEADER
I20260812 18:56:57.065408 2563 consensus_queue.cc:237] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:56:57.069603 2003 catalog_manager.cc:5495] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "9b25ab2635074f14b03cfed6e359d403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } health_report { overall_health: HEALTHY } } }
I20260812 18:56:57.088091 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.083994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.16.40.195:45421)
0812 18:56:57.084163 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 18:56:57.084167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:56:57.084184 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:56:57.084347 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:56:57.084350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:56:57.084429 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:56:57.084598 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:56:57.084877 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.085409 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.086354 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:56:57.086580 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:56:57.086659 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:56:57.087035 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:56:57.087606 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:56:57.087728 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 18:56:57.087864 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 18:56:57.087933 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 18:56:57.095527 2319 raft_consensus.cc:1261] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Refusing update from remote peer 9b25ab2635074f14b03cfed6e359d403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:57.095532 2166 raft_consensus.cc:1261] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Refusing update from remote peer 9b25ab2635074f14b03cfed6e359d403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:56:57.096426 2568 consensus_queue.cc:1035] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:57.096732 2563 consensus_queue.cc:1035] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:56:57.106424 2575 mvcc.cc:205] Tried to move back new op lower bound from 7317753926013640704 to 7317753925903503360. Current Snapshot: MvccSnapshot[applied={T|T < 7317753926013640704}]
I20260812 18:56:57.108340 2573 mvcc.cc:205] Tried to move back new op lower bound from 7317753926013640704 to 7317753925903503360. Current Snapshot: MvccSnapshot[applied={T|T < 7317753926013640704}]
I20260812 18:56:57.116389 2000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:57.116679 2000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:56:57.120020 2000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Sending DeleteTablet for 3 replicas of tablet 203e484ab20e486e950a83cb331fff8e
I20260812 18:56:57.121062 2295 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e484ab20e486e950a83cb331fff8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:57 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260812 18:56:57.121054 2455 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e484ab20e486e950a83cb331fff8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:57 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:33180
I20260812 18:56:57.121081 2143 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e484ab20e486e950a83cb331fff8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:57 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:58114
I20260812 18:56:57.121718 2583 tablet_replica.cc:330] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: stopping tablet replica
I20260812 18:56:57.121830 2585 tablet_replica.cc:330] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: stopping tablet replica
I20260812 18:56:57.121876 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2070
I20260812 18:56:57.137221 2584 tablet_replica.cc:330] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: stopping tablet replica
I20260812 18:56:57.137864 2584 raft_consensus.cc:2229] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:56:57.138507 2584 raft_consensus.cc:2258] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:57.140395 2585 raft_consensus.cc:2229] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:56:57.141045 2585 raft_consensus.cc:2258] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:56:57.143095 2585 ts_tablet_manager.cc:1905] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:57.147214 2585 ts_tablet_manager.cc:1918] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:57.147492 2585 log.cc:1196] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/203e484ab20e486e950a83cb331fff8e
I20260812 18:56:57.147993 2585 ts_tablet_manager.cc:1939] T 203e484ab20e486e950a83cb331fff8e P f0ba1ee9905e4553b54853fe95dda7a9: Deleting consensus metadata
I20260812 18:56:57.149830 1986 catalog_manager.cc:4841] TS f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): tablet 203e484ab20e486e950a83cb331fff8e (table test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]) successfully deleted
W20260812 18:56:57.150663 1984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:34591 (error 108)
I20260812 18:56:57.151851 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:56:57.151505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.0.0.1:58120)
0812 18:56:57.151661 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 18:56:57.151745 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 18:56:57.152029 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2223
W20260812 18:56:57.155746 1984 catalog_manager.cc:4584] TS 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 203e484ab20e486e950a83cb331fff8e: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34591: connect: Connection refused (error 111)
I20260812 18:56:57.161979 2584 ts_tablet_manager.cc:1905] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:56:57.165386 2584 ts_tablet_manager.cc:1918] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:56:57.165556 2584 log.cc:1196] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/203e484ab20e486e950a83cb331fff8e
I20260812 18:56:57.165891 2584 ts_tablet_manager.cc:1939] T 203e484ab20e486e950a83cb331fff8e P 9b25ab2635074f14b03cfed6e359d403: Deleting consensus metadata
I20260812 18:56:57.167259 1983 catalog_manager.cc:4841] TS 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): tablet 203e484ab20e486e950a83cb331fff8e (table test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]) successfully deleted
I20260812 18:56:57.173341 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2378
I20260812 18:56:57.184957 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 1963
I20260812 18:57:17.196756 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--webserver_interface=127.16.40.254
--webserver_port=40669
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:17.325771 2587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:17.326098 2587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:17.326143 2587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:17.330564 2587 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 18:57:17.330646 2587 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 18:57:17.330663 2587 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 18:57:17.330679 2587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:57:17.330699 2587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:17.330763 2587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:17.330794 2587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:57:17.330827 2587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:57:17.336869 2587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:17.338163 2587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:17.339386 2587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:17.355538 2597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:17.355561 2595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:17.355597 2594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:17.356004 2587 server_base.cc:1034] running on GCE node
I20260812 18:57:17.356813 2587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:17.358374 2587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:17.359653 2587 hybrid_clock.cc:648] HybridClock initialized: now 1786561037359636 us; error 49 us; skew 500 ppm
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:17.369234 2587 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:57:17.370450 2587 webserver.cc:460] Webserver started at http://127.16.40.254:40669/ using document root <none> and password file <none>
I20260812 18:57:17.371150 2587 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:17.371266 2587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:17.375900 2587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:17.378643 2605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:17.380051 2587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 18:57:17.380211 2587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a7cb6254fd15439bbe7bd197bed76448"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "68eda0b4f8fa679a85795d594fe03561"
server_key_iv: "066e2494ce7e3e6c812fe23ac0975005"
server_key_version: "encryptionkey@0"
I20260812 18:57:17.380921 2587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:17.402344 2587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:17.406263 2587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:17.406570 2587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:17.414924 2587 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:40205
I20260812 18:57:17.415253 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:40205 every 8 connection(s)
I20260812 18:57:17.416396 2587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:57:17.421788 2667 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:57:17.423870 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap starting.
I20260812 18:57:17.426352 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2587
I20260812 18:57:17.426932 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:34591
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=32885
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:57:17.443892 2667 log.cc:824] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:17.454110 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:57:17.454607 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap complete.
I20260812 18:57:17.459072 2667 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:17.459533 2667 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Initialized, Role: FOLLOWER
I20260812 18:57:17.460078 2667 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:17.460206 2667 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:57:17.460306 2667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:57:17.460426 2667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 1 FOLLOWER]: Advancing to term 2
I20260812 18:57:17.462754 2667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:17.463148 2667 leader_election.cc:302] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7cb6254fd15439bbe7bd197bed76448; no voters:
I20260812 18:57:17.463554 2667 leader_election.cc:288] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 18:57:17.463656 2671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 18:57:17.463866 2671 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 LEADER]: Becoming Leader. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Running, Role: LEADER
I20260812 18:57:17.464282 2671 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:17.465042 2667 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:57:17.465787 2672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:57:17.465922 2672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:17.466672 2673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cb6254fd15439bbe7bd197bed76448. Latest consensus state: current_term: 2 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:57:17.466780 2673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:17.466948 2677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:57:17.472326 2677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:57:17.473515 2677 catalog_manager.cc:1259] Loaded cluster ID: 88b7c5e4848549e59cb86815cca41fdb
I20260812 18:57:17.473600 2677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:57:17.476272 2677 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:57:17.477216 2677 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Loaded TSK: 0
I20260812 18:57:17.493826 2677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Generated new TSK 1
I20260812 18:57:17.494182 2677 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:57:17.556262 2669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:17.556658 2669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:17.556751 2669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:17.561125 2669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:17.561201 2669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:17.561290 2669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:57:17.567677 2669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:34591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=32885
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:17.569031 2669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:17.570173 2669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:17.581975 2700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:17.582175 2669 server_base.cc:1034] running on GCE node
W20260812 18:57:17.582000 2704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:17.582371 2701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:17.582871 2669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:17.583789 2669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:17.584987 2669 hybrid_clock.cc:648] HybridClock initialized: now 1786561037584944 us; error 54 us; skew 500 ppm
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:17.588979 2669 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:57:17.590394 2669 webserver.cc:460] Webserver started at http://127.16.40.193:32885/ using document root <none> and password file <none>
I20260812 18:57:17.591226 2669 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:17.591359 2669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:17.597100 2669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:57:17.600189 2713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:17.601994 2669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:17.602149 2669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "4f20b5b385f946718d32186057ad5913"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "cf9874ef9ac00e4057e7a02b9ef81353"
server_key_iv: "c360b538b88228549974089e4eda6bf3"
server_key_version: "encryptionkey@0"
I20260812 18:57:17.602941 2669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:17.628412 2669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:17.632489 2669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:17.632752 2669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:17.633479 2669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:17.634855 2721 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 18:57:17.639473 2669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 18:57:17.639559 2669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 18:57:17.639647 2669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 18:57:17.641558 2669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 18:57:17.641623 2669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260812 18:57:17.641976 2721 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Bootstrap starting.
I20260812 18:57:17.653122 2669 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:34591
I20260812 18:57:17.653232 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:34591 every 8 connection(s)
I20260812 18:57:17.654258 2669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:57:17.656484 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2669
I20260812 18:57:17.657029 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:33735
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=37281
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:57:17.663002 2721 log.cc:824] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:17.674351 2721 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:57:17.674427 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.656986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:45181 (local address 127.16.40.254:40205)
0812 18:57:17.657583 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:17.657610 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:17.659919 (+ 2309us) server_negotiation.cc:407] Connection header received
0812 18:57:17.660172 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:17.660182 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:17.660511 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:17.660950 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:17.662189 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.663315 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.664949 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.665556 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.665787 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.668796 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:17.668802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:17.668817 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:17.668849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:17.670828 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.671444 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.671456 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.671460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.671593 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.671969 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.671973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.671974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.672403 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:17.672595 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:17.673197 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 18:57:17.673498 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":127,"threads_started":1}
I20260812 18:57:17.674443 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.656993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:45181)
0812 18:57:17.658548 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:17.658633 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:17.659634 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:17.661016 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:17.661025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:17.661491 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:17.661985 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.662001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.663508 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.663515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.664798 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.664812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.665802 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.665808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.666062 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.666806 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:17.666835 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:17.668522 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:17.670983 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.670990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.671006 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.671301 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.671703 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.671706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.671708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.671862 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.672572 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:17.672580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:17.672848 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 18:57:17.673043 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 18:57:17.673451 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":176,"threads_started":1}
I20260812 18:57:17.674991 2721 tablet_bootstrap.cc:492] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Bootstrap complete.
I20260812 18:57:17.676828 2721 ts_tablet_manager.cc:1397] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.008s
I20260812 18:57:17.677600 2844 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:57:17.677856 2844 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:17.678432 2844 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:57:17.680157 2721 raft_consensus.cc:348] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:57:17.680788 2721 raft_consensus.cc:729] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f20b5b385f946718d32186057ad5913, State: Initialized, Role: FOLLOWER
I20260812 18:57:17.681211 2622 ts_manager.cc:194] Registered new tserver with Master: 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:57:17.681350 2721 consensus_queue.cc:260] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } }
I20260812 18:57:17.682622 2721 ts_tablet_manager.cc:1428] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:57:17.685997 2622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:45181
I20260812 18:57:17.688828 2844 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.193@KRBTEST.COM
I20260812 18:57:17.697497 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.684840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.0.0.1:57806)
0812 18:57:17.685351 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:17.685414 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:17.685573 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:17.686504 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:17.686511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:17.687029 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:17.687178 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.687188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.688657 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.688661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.689555 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.689589 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.690370 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.690373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.690494 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.691174 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:17.691187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:17.692786 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:17.695230 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.695240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.695246 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.695524 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.695902 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.695905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.695907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.696015 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.696728 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:17.696734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:17.696874 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 18:57:17.697028 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 18:57:17.697294 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":152,"threads_started":1}
I20260812 18:57:17.697497 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.684917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.16.40.193:34591)
0812 18:57:17.685982 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:17.685994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:17.686020 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:57:17.686100 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:17.686105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:17.686327 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:17.686472 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:17.687328 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.688482 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.689716 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.690233 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.690347 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.692966 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:17.692980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:17.692995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:17.693041 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:17.695079 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.695657 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.695663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.695665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.695776 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.696193 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.696196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.696198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.696488 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:17.696695 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:17.697127 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 18:57:17.697313 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":111,"threads_started":1}
I20260812 18:57:17.698881 2767 tablet_service.cc:1514] Processing DeleteTablet for tablet 203e484ab20e486e950a83cb331fff8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:56:57 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57806
I20260812 18:57:17.700057 2855 tablet_replica.cc:330] stopping tablet replica
I20260812 18:57:17.700392 2855 raft_consensus.cc:2229] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:17.700719 2855 raft_consensus.cc:2258] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:17.701785 2855 ts_tablet_manager.cc:1905] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:17.705807 2855 ts_tablet_manager.cc:1918] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:17.705956 2855 log.cc:1196] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal/wals/203e484ab20e486e950a83cb331fff8e
I20260812 18:57:17.706231 2855 ts_tablet_manager.cc:1939] T 203e484ab20e486e950a83cb331fff8e P 4f20b5b385f946718d32186057ad5913: Deleting consensus metadata
I20260812 18:57:17.708827 2607 catalog_manager.cc:4841] TS 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591): tablet 203e484ab20e486e950a83cb331fff8e (table test-table [id=7dc42d1bf17c42d38fe3608235a7dd64]) successfully deleted
W20260812 18:57:17.794343 2848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:17.794674 2848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:17.794721 2848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:17.799060 2848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:17.799141 2848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:17.799213 2848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:57:17.805040 2848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:33735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=37281
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:17.806227 2848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:17.807268 2848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:17.818050 2865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:17.817961 2863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:17.818495 2848 server_base.cc:1034] running on GCE node
W20260812 18:57:17.818684 2862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:17.819182 2848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:17.819914 2848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:17.821122 2848 hybrid_clock.cc:648] HybridClock initialized: now 1786561037821080 us; error 57 us; skew 500 ppm
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:17.824323 2848 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:57:17.825564 2848 webserver.cc:460] Webserver started at http://127.16.40.194:37281/ using document root <none> and password file <none>
I20260812 18:57:17.826229 2848 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:17.826350 2848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:17.830847 2848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:57:17.833356 2873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:17.834995 2848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:17.835173 2848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "7ef4cb5d75f93066e74262909bb8f57c"
server_key_iv: "21552df2342f9c5afd494106a76baa0f"
server_key_version: "encryptionkey@0"
I20260812 18:57:17.835790 2848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:17.858647 2848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:17.862246 2848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:17.862468 2848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:17.863111 2848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:17.864017 2848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:17.864091 2848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:17.864159 2848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:17.864205 2848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:17.874387 2848 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33735
I20260812 18:57:17.874410 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33735 every 8 connection(s)
I20260812 18:57:17.875550 2848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:57:17.879967 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 2848
I20260812 18:57:17.880502 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:45421
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=45049
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 18:57:17 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561037, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:17.892277 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.877747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:56293 (local address 127.16.40.254:40205)
0812 18:57:17.877928 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:17.877934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:17.878789 (+ 855us) server_negotiation.cc:407] Connection header received
0812 18:57:17.880227 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:17.880231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:17.880304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:17.880407 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:17.881807 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.882569 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.883952 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.884405 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.884559 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.887634 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:17.887639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:17.887644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:17.887684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:17.889642 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.890273 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.890278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.890282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.890405 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:17.890784 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:17.890789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:17.890793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:17.891110 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:17.891233 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:17.891822 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 18:57:17.892017 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:57:17.893118 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:17.878139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:56293)
0812 18:57:17.878627 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:17.878675 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:17.879983 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:17.880665 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:17.880676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:17.881115 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:17.881641 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.881655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.882738 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.882741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.883811 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:17.883822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:17.884701 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:17.884707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:17.884895 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:17.885645 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:17.885672 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:17.887368 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:17.889789 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.889796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.889812 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.890127 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.890513 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:17.890515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:17.890516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:17.890681 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:17.891251 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:17.891257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:17.891539 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 18:57:17.891730 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 18:57:17.892045 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260812 18:57:17.894588 3005 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:57:17.894881 3005 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:17.895537 3005 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:57:17.896826 2622 ts_manager.cc:194] Registered new tserver with Master: f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735)
I20260812 18:57:17.897446 2622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:56293
W20260812 18:57:18.007421 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:18.007736 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:18.007784 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:18.012185 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:18.012284 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:18.012349 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:57:18.018518 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:45421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=45049
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:18.019675 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:18.020835 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:18.031339 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:18.031360 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:18.031339 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:18.031746 3009 server_base.cc:1034] running on GCE node
I20260812 18:57:18.032181 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:18.033113 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:18.034438 3009 hybrid_clock.cc:648] HybridClock initialized: now 1786561038034414 us; error 37 us; skew 500 ppm
Aug 12 18:57:18 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561038, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:18.037606 3009 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:57:18.038618 3009 webserver.cc:460] Webserver started at http://127.16.40.195:45049/ using document root <none> and password file <none>
I20260812 18:57:18.039235 3009 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:18.039364 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:18.043864 3009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:18.046398 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:18.048028 3009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:18.048208 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "9b25ab2635074f14b03cfed6e359d403"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "2e3b6c50930ef1716f776cadf59637b4"
server_key_iv: "239008b0b8ed8507714bb5a3bfc5e530"
server_key_version: "encryptionkey@0"
I20260812 18:57:18.048930 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:18.061869 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:18.065465 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:18.065716 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:18.066488 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:18.067502 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:18.067574 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:18.067653 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:18.067698 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:18.078027 3009 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:45421
I20260812 18:57:18.078037 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:45421 every 8 connection(s)
I20260812 18:57:18.079219 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:57:18.079456 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3009
I20260812 18:57:18.089659 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.080264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.16.40.254:40205)
0812 18:57:18.080429 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.080434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.080854 (+ 420us) server_negotiation.cc:407] Connection header received
0812 18:57:18.081015 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.081018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.081072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.081154 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:18.082123 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.082775 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.083691 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.084002 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.084171 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.085824 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:18.085828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:18.085831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:18.085862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:18.087468 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.087984 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.087989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.088007 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.088130 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.088510 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.088513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.088515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.088942 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:18.089196 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.089336 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.089473 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 18:57:18 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561038, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:18.096768 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.081382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:37767 (local address 127.16.40.254:40205)
0812 18:57:18.081635 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.081638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.082793 (+ 1155us) server_negotiation.cc:407] Connection header received
0812 18:57:18.084370 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.084374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.084430 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.084532 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:18.086117 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.086881 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.088734 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.089021 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.089148 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.092472 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:18.092477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:18.092481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:18.092517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:18.094241 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.094880 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.094885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.094887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.094978 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.095400 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.095405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.095407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.095640 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:18.095762 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.096343 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.096497 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":115,"threads_started":1}
I20260812 18:57:18.097803 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.082040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:37767)
0812 18:57:18.082582 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.082639 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:18.084157 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:18.084774 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:18.084787 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:18.085314 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:18.085967 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.085985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.087033 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.087036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.088599 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.088614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.089204 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.089209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.089397 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.090357 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:18.090387 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:18.092271 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:18.094394 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:18.094402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:18.094422 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:18.094756 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:18.095111 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:18.095115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:18.095117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:18.095292 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:18.095759 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:18.095767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:18.096074 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 18:57:18.096298 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 18:57:18.096795 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260812 18:57:18.099187 3156 heartbeater.cc:347] Connected to a master server at 127.16.40.254:40205
I20260812 18:57:18.099474 3156 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:18.100072 3156 heartbeater.cc:511] Master 127.16.40.254:40205 requested a full tablet report, sending...
I20260812 18:57:18.101272 2622 ts_manager.cc:194] Registered new tserver with Master: 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
I20260812 18:57:18.101886 2622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:37767
I20260812 18:57:18.103878 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:57:18.107967 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.104513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.16.40.254:40205)
0812 18:57:18.104693 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.104697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.104868 (+ 171us) server_negotiation.cc:407] Connection header received
0812 18:57:18.104996 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.104999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.105096 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.105185 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:57:18.105387 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.105891 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.106752 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.107039 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.107145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.107456 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:57:18.107610 (+ 154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 18:57:18.107852 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57882: token expired
Metrics: {"server-negotiator.queue_time_us":104}
W20260812 18:57:18.108067 3160 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57882: token expired
W20260812 18:57:18.108462 2537 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.40.254:40205)
I20260812 18:57:18.116748 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.108897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.16.40.254:40205)
0812 18:57:18.109015 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.109022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.109054 (+ 32us) server_negotiation.cc:407] Connection header received
0812 18:57:18.109207 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.109210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.109253 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.109336 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:18.110055 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.110560 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.111396 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.111641 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.111749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.113190 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:18.113194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:18.113197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:18.113227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:18.114892 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.115379 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.115383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.115385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.115488 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:18.115791 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:18.115795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:18.115797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:18.116009 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:18.116150 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.116400 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.116583 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 18:57:18.120052 2622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:57:18.123767 2622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:57:18.133049 2767 tablet_service.cc:1467] Processing CreateTablet for tablet e82e43d4b9984fae846332b16603194f (DEFAULT_TABLE table=test-table [id=c57ec18268de4a39b0d18e6a29439678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:18.133834 2767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82e43d4b9984fae846332b16603194f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:18.137209 2721 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Bootstrap starting.
I20260812 18:57:18.137902 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.131877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33735 (local address 127.0.0.1:51872)
0812 18:57:18.132171 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.132190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:18.132271 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:18.133348 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:18.133351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:18.133373 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:18.133523 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.133533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.134958 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.134962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.136148 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.136159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.137394 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.137398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.137499 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.137561 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 18:57:18.137630 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 18:57:18.137752 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 18:57:18.138087 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.132349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:45421 (local address 127.0.0.1:51894)
0812 18:57:18.133387 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.133402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:18.133493 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:18.134174 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:18.134177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:18.134195 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:18.134271 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.134276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.135793 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.135797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.136615 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.136626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.137725 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.137729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.137808 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.137830 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 18:57:18.137890 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 18:57:18.137964 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":142,"threads_started":1}
I20260812 18:57:18.138332 2721 tablet_bootstrap.cc:654] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:18.138609 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.132871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.16.40.195:45421)
0812 18:57:18.133240 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.133253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.133594 (+ 341us) server_negotiation.cc:407] Connection header received
0812 18:57:18.133686 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.133690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.133840 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.134012 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:18.134398 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.135663 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.136738 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.137582 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.137717 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.137881 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.137942 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.138434 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":161,"threads_started":1}
I20260812 18:57:18.138717 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.132428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.16.40.194:33735)
0812 18:57:18.132813 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.132827 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.132852 (+ 25us) server_negotiation.cc:407] Connection header received
0812 18:57:18.132937 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.132943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.133185 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.133543 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:18.133749 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.134811 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.136311 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.137237 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.138213 (+ 976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.138383 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.138459 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.138559 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":144,"threads_started":1}
I20260812 18:57:18.140326 2721 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: No bootstrap required, opened a new log
I20260812 18:57:18.140442 2721 ts_tablet_manager.cc:1397] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260812 18:57:18.141134 2721 raft_consensus.cc:348] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.141264 2721 raft_consensus.cc:374] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:18.141312 2721 raft_consensus.cc:729] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f20b5b385f946718d32186057ad5913, State: Initialized, Role: FOLLOWER
I20260812 18:57:18.141484 2721 consensus_queue.cc:260] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.141782 2721 ts_tablet_manager.cc:1428] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 18:57:18.142197 3079 tablet_service.cc:1467] Processing CreateTablet for tablet e82e43d4b9984fae846332b16603194f (DEFAULT_TABLE table=test-table [id=c57ec18268de4a39b0d18e6a29439678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:18.142194 2927 tablet_service.cc:1467] Processing CreateTablet for tablet e82e43d4b9984fae846332b16603194f (DEFAULT_TABLE table=test-table [id=c57ec18268de4a39b0d18e6a29439678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:18.143466 2927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82e43d4b9984fae846332b16603194f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:18.143466 3079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82e43d4b9984fae846332b16603194f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:18.153707 3172 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Bootstrap starting.
W20260812 18:57:18.155578 2845 tablet.cc:2369] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:57:18.156344 3172 tablet_bootstrap.cc:654] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:18.156991 3171 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Bootstrap starting.
I20260812 18:57:18.157428 3172 log.cc:824] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:18.160004 3172 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: No bootstrap required, opened a new log
I20260812 18:57:18.160034 3171 tablet_bootstrap.cc:654] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:18.160283 3172 ts_tablet_manager.cc:1397] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 18:57:18.161042 3171 log.cc:824] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:18.163370 3171 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: No bootstrap required, opened a new log
I20260812 18:57:18.163597 3171 ts_tablet_manager.cc:1397] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260812 18:57:18.164454 3172 raft_consensus.cc:348] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.164815 3172 raft_consensus.cc:374] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:18.164882 3172 raft_consensus.cc:729] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ba1ee9905e4553b54853fe95dda7a9, State: Initialized, Role: FOLLOWER
I20260812 18:57:18.165390 3172 consensus_queue.cc:260] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.166177 3005 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
I20260812 18:57:18.166550 3172 ts_tablet_manager.cc:1428] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:57:18.167688 3171 raft_consensus.cc:348] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.167922 3171 raft_consensus.cc:374] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:18.167979 3171 raft_consensus.cc:729] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b25ab2635074f14b03cfed6e359d403, State: Initialized, Role: FOLLOWER
I20260812 18:57:18.168416 3171 consensus_queue.cc:260] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.169212 3156 heartbeater.cc:503] Master 127.16.40.254:40205 was elected leader, sending a full tablet report...
I20260812 18:57:18.169510 3171 ts_tablet_manager.cc:1428] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260812 18:57:18.201767 3175 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:18.202015 3175 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.203557 3175 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421), 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:57:18.208619 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.204032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:55671 (local address 127.16.40.195:45421)
0812 18:57:18.204264 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.204269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.204285 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:57:18.204340 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.204343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.204392 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.204482 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:18.204822 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.205579 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.207270 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.207944 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.208090 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.208153 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.208360 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.208442 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260812 18:57:18.208772 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.203865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.16.40.194:60357)
0812 18:57:18.204325 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.204339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:18.204430 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:18.204935 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:18.204937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:18.204954 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:18.205059 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.205064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.205963 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.205967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.207278 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.207287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.208339 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.208343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.208448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.208472 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:57:18.208564 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 18:57:18.208631 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"thread_start_us":136,"threads_started":1}
I20260812 18:57:18.208649 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.203755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:45421 (local address 127.16.40.194:55671)
0812 18:57:18.203889 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.203906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:18.204052 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:18.204524 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:18.204530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:18.204575 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:18.204706 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.204712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.205702 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.205705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.207100 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:18.207113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.208092 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.208095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:18.208193 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.208218 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:57:18.208327 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 18:57:18.208419 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 18:57:18.208962 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.204340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:60357 (local address 127.16.40.193:34591)
0812 18:57:18.204652 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.204656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.204668 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:57:18.204713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.204715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.204788 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.204907 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:18.205177 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.205845 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.207424 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.208203 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.208334 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.208634 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.208689 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.208771 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260812 18:57:18.209594 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82e43d4b9984fae846332b16603194f" candidate_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f20b5b385f946718d32186057ad5913" is_pre_election: true
I20260812 18:57:18.209704 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82e43d4b9984fae846332b16603194f" candidate_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b25ab2635074f14b03cfed6e359d403" is_pre_election: true
I20260812 18:57:18.210006 3105 raft_consensus.cc:2454] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ba1ee9905e4553b54853fe95dda7a9 in term 0.
I20260812 18:57:18.210008 2791 raft_consensus.cc:2454] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0ba1ee9905e4553b54853fe95dda7a9 in term 0.
I20260812 18:57:18.210588 2874 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f20b5b385f946718d32186057ad5913, 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9; no voters:
I20260812 18:57:18.210929 3175 raft_consensus.cc:2781] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:57:18.211035 3175 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:57:18.211093 3175 raft_consensus.cc:3037] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:18.212350 3175 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.212852 3175 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [CANDIDATE]: Term 1 election: Requested vote from peers 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421), 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591)
I20260812 18:57:18.213335 3105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82e43d4b9984fae846332b16603194f" candidate_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b25ab2635074f14b03cfed6e359d403"
I20260812 18:57:18.213500 3105 raft_consensus.cc:3037] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:18.214947 3105 raft_consensus.cc:2454] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ba1ee9905e4553b54853fe95dda7a9 in term 1.
I20260812 18:57:18.215245 2791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82e43d4b9984fae846332b16603194f" candidate_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f20b5b385f946718d32186057ad5913"
I20260812 18:57:18.215420 2874 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9; no voters:
I20260812 18:57:18.215444 2791 raft_consensus.cc:3037] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:18.215688 3175 raft_consensus.cc:2781] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:18.215934 3175 raft_consensus.cc:686] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 LEADER]: Becoming Leader. State: Replica: f0ba1ee9905e4553b54853fe95dda7a9, State: Running, Role: LEADER
I20260812 18:57:18.216358 3175 consensus_queue.cc:237] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:18.216871 2791 raft_consensus.cc:2454] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ba1ee9905e4553b54853fe95dda7a9 in term 1.
I20260812 18:57:18.220206 2622 catalog_manager.cc:5495] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } health_report { overall_health: UNKNOWN } } }
I20260812 18:57:18.249984 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.246059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.16.40.194:33735)
0812 18:57:18.246212 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:18.246216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:18.246230 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:57:18.246376 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:18.246378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:18.246440 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:18.246535 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:57:18.246806 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.247338 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.248290 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:18.248513 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:18.248636 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:18.249029 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:57:18.249531 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:57:18.249638 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:18.249751 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 18:57:18.249817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 18:57:18.257385 3105 raft_consensus.cc:1261] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Refusing update from remote peer f0ba1ee9905e4553b54853fe95dda7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:18.257380 2791 raft_consensus.cc:1261] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Refusing update from remote peer f0ba1ee9905e4553b54853fe95dda7a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:18.258191 3175 consensus_queue.cc:1035] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:18.258461 3181 consensus_queue.cc:1035] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:18.267010 3185 mvcc.cc:205] Tried to move back new op lower bound from 7317754012692623360 to 7317754012537843712. Current Snapshot: MvccSnapshot[applied={T|T < 7317754012692623360}]
I20260812 18:57:18.268443 3184 mvcc.cc:205] Tried to move back new op lower bound from 7317754012692623360 to 7317754012537843712. Current Snapshot: MvccSnapshot[applied={T|T < 7317754012692623360}]
I20260812 18:57:18.279085 2621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:18.279268 2621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:18.281919 2621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Sending DeleteTablet for 3 replicas of tablet e82e43d4b9984fae846332b16603194f
I20260812 18:57:18.282757 2767 tablet_service.cc:1514] Processing DeleteTablet for tablet e82e43d4b9984fae846332b16603194f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:18 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57806
I20260812 18:57:18.282850 2927 tablet_service.cc:1514] Processing DeleteTablet for tablet e82e43d4b9984fae846332b16603194f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:18 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:51872
I20260812 18:57:18.283469 3196 tablet_replica.cc:330] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: stopping tablet replica
I20260812 18:57:18.283440 3079 tablet_service.cc:1514] Processing DeleteTablet for tablet e82e43d4b9984fae846332b16603194f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:18 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:51894
I20260812 18:57:18.283689 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2669
I20260812 18:57:18.283882 3196 raft_consensus.cc:2229] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:57:18.284485 3196 raft_consensus.cc:2258] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:18.287177 3196 ts_tablet_manager.cc:1905] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:18.288029 3197 tablet_replica.cc:330] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: stopping tablet replica
I20260812 18:57:18.288435 3197 raft_consensus.cc:2229] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:18.288919 3197 raft_consensus.cc:2258] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:18.290797 3197 ts_tablet_manager.cc:1905] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:18.291091 3196 ts_tablet_manager.cc:1918] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:18.291263 3196 log.cc:1196] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/e82e43d4b9984fae846332b16603194f
I20260812 18:57:18.291759 3196 ts_tablet_manager.cc:1939] T e82e43d4b9984fae846332b16603194f P f0ba1ee9905e4553b54853fe95dda7a9: Deleting consensus metadata
I20260812 18:57:18.293973 3197 ts_tablet_manager.cc:1918] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:18.294129 3197 log.cc:1196] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/e82e43d4b9984fae846332b16603194f
I20260812 18:57:18.294490 3197 ts_tablet_manager.cc:1939] T e82e43d4b9984fae846332b16603194f P 9b25ab2635074f14b03cfed6e359d403: Deleting consensus metadata
I20260812 18:57:18.295835 2606 catalog_manager.cc:4841] TS 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): tablet e82e43d4b9984fae846332b16603194f (table test-table [id=c57ec18268de4a39b0d18e6a29439678]) successfully deleted
I20260812 18:57:18.296763 2608 catalog_manager.cc:4841] TS f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): tablet e82e43d4b9984fae846332b16603194f (table test-table [id=c57ec18268de4a39b0d18e6a29439678]) successfully deleted
W20260812 18:57:18.296927 2607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:34591 (error 108)
I20260812 18:57:18.298062 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:18.297731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:34591 (local address 127.0.0.1:57812)
0812 18:57:18.297877 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:18.297980 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 18:57:18.298262 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2848
W20260812 18:57:18.302399 2607 catalog_manager.cc:4584] TS 4f20b5b385f946718d32186057ad5913 (127.16.40.193:34591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e82e43d4b9984fae846332b16603194f: Network error: Client connection negotiation failed: client connection to 127.16.40.193:34591: connect: Connection refused (error 111)
I20260812 18:57:18.308391 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3009
I20260812 18:57:18.317862 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 2587
I20260812 18:57:38.328512 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--webserver_interface=127.16.40.254
--webserver_port=40669
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:38.453608 3199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:38.453985 3199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:38.454066 3199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:38.458549 3199 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 18:57:38.458660 3199 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 18:57:38.458688 3199 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 18:57:38.458711 3199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:57:38.458755 3199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:38.458819 3199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:38.458860 3199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:57:38.458894 3199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:57:38.464879 3199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:40205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=40669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:38.466156 3199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:38.467304 3199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:38.477366 3204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:38.477409 3206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:38.477483 3199 server_base.cc:1034] running on GCE node
W20260812 18:57:38.477705 3208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:38.478430 3199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:38.479669 3199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:38.480841 3199 hybrid_clock.cc:648] HybridClock initialized: now 1786561058480806 us; error 62 us; skew 500 ppm
Aug 12 18:57:38 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:38.490094 3199 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:57:38.491245 3199 webserver.cc:460] Webserver started at http://127.16.40.254:40669/ using document root <none> and password file <none>
I20260812 18:57:38.491870 3199 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:38.491945 3199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:38.496989 3199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:57:38.499711 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:38.501526 3199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:38.501719 3199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a7cb6254fd15439bbe7bd197bed76448"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "68eda0b4f8fa679a85795d594fe03561"
server_key_iv: "066e2494ce7e3e6c812fe23ac0975005"
server_key_version: "encryptionkey@0"
I20260812 18:57:38.502418 3199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:38.523197 3199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:38.527555 3199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:38.527868 3199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:38.537463 3199 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:40205
I20260812 18:57:38.537782 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:40205 every 8 connection(s)
I20260812 18:57:38.538965 3199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:57:38.544601 3278 sys_catalog.cc:263] Verifying existing consensus state
I20260812 18:57:38.546595 3278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap starting.
I20260812 18:57:38.548175 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3199
I20260812 18:57:38.548760 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:34591
--local_ip_for_outbound_sockets=127.16.40.193
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=32885
--webserver_interface=127.16.40.193
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:57:38.565768 3278 log.cc:824] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:38.578123 3278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:57:38.578591 3278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Bootstrap complete.
I20260812 18:57:38.583039 3278 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:38.583550 3278 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Initialized, Role: FOLLOWER
I20260812 18:57:38.584199 3278 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:38.584368 3278 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:57:38.584467 3278 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:57:38.584637 3278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 2 FOLLOWER]: Advancing to term 3
I20260812 18:57:38.586818 3278 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:38.587219 3278 leader_election.cc:302] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7cb6254fd15439bbe7bd197bed76448; no voters:
I20260812 18:57:38.587611 3278 leader_election.cc:288] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 18:57:38.587779 3283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 18:57:38.588060 3283 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [term 3 LEADER]: Becoming Leader. State: Replica: a7cb6254fd15439bbe7bd197bed76448, State: Running, Role: LEADER
I20260812 18:57:38.588493 3283 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } }
I20260812 18:57:38.589080 3278 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:57:38.589802 3283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cb6254fd15439bbe7bd197bed76448. Latest consensus state: current_term: 3 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:57:38.589927 3283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:38.590289 3284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a7cb6254fd15439bbe7bd197bed76448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cb6254fd15439bbe7bd197bed76448" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40205 } } }
I20260812 18:57:38.590474 3289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:57:38.590478 3284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:38.596750 3289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:57:38.597640 3289 catalog_manager.cc:1259] Loaded cluster ID: 88b7c5e4848549e59cb86815cca41fdb
I20260812 18:57:38.597744 3289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:57:38.599674 3289 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:57:38.600394 3289 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Loaded TSK: 0
I20260812 18:57:38.600615 3289 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a7cb6254fd15439bbe7bd197bed76448: Loaded TSK: 1
F20260812 18:57:38.601326 3289 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f58f88a49ad google::LogMessage::Fail() at ??:0
@ 0x7f58f88a788a google::LogMessage::SendToLog() at ??:0
@ 0x7f58f88a44de google::LogMessage::Flush() at ??:0
@ 0x7f58f88a5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f58fb56818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f58fb560f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f59032097bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f59031e399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f59031e4178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f5903225df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f590324482b std::function<>::operator()() at ??:0
@ 0x7f59031e3b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f59031e4f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f59031e1579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5903225640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f58faaf63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f58faaf6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f58faaf83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f58faae7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f58fc12b6db start_thread at ??:0
@ 0x7f58f73a261f clone at ??:0
W20260812 18:57:38.716125 3281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:38.716444 3281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:38.716492 3281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:38.720934 3281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:38.721014 3281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:38.721115 3281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:57:38.726910 3281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:34591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=32885
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:38.728160 3281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:38.729331 3281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:38.740027 3330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:38.740150 3331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:38.740021 3333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:38.740288 3281 server_base.cc:1034] running on GCE node
I20260812 18:57:38.740849 3281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:38.741605 3281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:38.742820 3281 hybrid_clock.cc:648] HybridClock initialized: now 1786561058742772 us; error 65 us; skew 500 ppm
Aug 12 18:57:38 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:38.745880 3281 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:57:38.746951 3281 webserver.cc:460] Webserver started at http://127.16.40.193:32885/ using document root <none> and password file <none>
I20260812 18:57:38.747591 3281 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:38.747663 3281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:38.751959 3281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:38.754395 3343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:38.755806 3281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:38.755919 3281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "4f20b5b385f946718d32186057ad5913"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "cf9874ef9ac00e4057e7a02b9ef81353"
server_key_iv: "c360b538b88228549974089e4eda6bf3"
server_key_version: "encryptionkey@0"
I20260812 18:57:38.756521 3281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:38.767969 3281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:38.771492 3281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:38.771711 3281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:38.772398 3281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:38.773624 3352 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 18:57:38.779072 3281 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 18:57:38.779179 3281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 18:57:38.779266 3281 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 18:57:38.781828 3281 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 18:57:38.781905 3281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:38.782423 3352 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Bootstrap starting.
I20260812 18:57:38.793964 3281 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:34591
I20260812 18:57:38.795260 3281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:57:38.796867 3482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:34591 every 8 connection(s)
I20260812 18:57:38.797963 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3281
I20260812 18:57:38.798473 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:33735
--local_ip_for_outbound_sockets=127.16.40.194
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=37281
--webserver_interface=127.16.40.194
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:57:38.807169 3352 log.cc:824] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:38.818500 3352 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 18:57:38.819020 3352 tablet_bootstrap.cc:492] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Bootstrap complete.
I20260812 18:57:38.820600 3352 ts_tablet_manager.cc:1397] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.000s
I20260812 18:57:38.824131 3352 raft_consensus.cc:348] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:38.824751 3352 raft_consensus.cc:729] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f20b5b385f946718d32186057ad5913, State: Initialized, Role: FOLLOWER
I20260812 18:57:38.825322 3352 consensus_queue.cc:260] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:38.826452 3352 ts_tablet_manager.cc:1428] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260812 18:57:38.848682 3485 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:57:38.797754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:48577)
0812 18:57:38.800946 (+ 3192us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:38.801036 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:38.802259 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:38.847783 (+ 45524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2889,"thread_start_us":157,"threads_started":1}
W20260812 18:57:38.849241 3483 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:40205 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:40205: Client connection negotiation failed: client connection to 127.16.40.254:40205: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 18:57:38.850076 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:38.849804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:37191)
0812 18:57:38.849948 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:38.850017 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 18:57:38.850747 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:38.850527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:60587)
0812 18:57:38.850645 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:38.850698 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 18:57:38.850917 3483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:57:38.932120 3486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:38.932489 3486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:38.932622 3486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:38.937005 3486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:38.937109 3486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:38.937207 3486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:57:38.943044 3486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:33735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=37281
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:38.944316 3486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:38.945513 3486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:38.956077 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:38.956123 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:38.956269 3486 server_base.cc:1034] running on GCE node
W20260812 18:57:38.956123 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:38.956908 3486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:38.957660 3486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:38.958853 3486 hybrid_clock.cc:648] HybridClock initialized: now 1786561058958823 us; error 41 us; skew 500 ppm
Aug 12 18:57:38 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:38.962224 3486 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:57:38.963294 3486 webserver.cc:460] Webserver started at http://127.16.40.194:37281/ using document root <none> and password file <none>
I20260812 18:57:38.963933 3486 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:38.964028 3486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:38.968281 3486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:38.970836 3506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:38.972363 3486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:57:38.972589 3486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "f0ba1ee9905e4553b54853fe95dda7a9"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "7ef4cb5d75f93066e74262909bb8f57c"
server_key_iv: "21552df2342f9c5afd494106a76baa0f"
server_key_version: "encryptionkey@0"
I20260812 18:57:38.973253 3486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:39.011116 3486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:39.014317 3486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:39.014555 3486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:39.015230 3486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:39.016129 3486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:39.016178 3486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:39.016248 3486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:39.016287 3486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:39.025960 3486 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33735
I20260812 18:57:39.025993 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33735 every 8 connection(s)
I20260812 18:57:39.027119 3486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:57:39.031368 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3486
I20260812 18:57:39.031514 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.029505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:56633)
0812 18:57:39.030085 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.030202 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":130,"threads_started":1}
I20260812 18:57:39.031924 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:45421
--local_ip_for_outbound_sockets=127.16.40.195
--tserver_master_addrs=127.16.40.254:40205
--webserver_port=45049
--webserver_interface=127.16.40.195
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:39.032183 3639 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:40205 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:40205: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
I20260812 18:57:39.033206 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.032918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:54493)
0812 18:57:39.033054 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.033141 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 18:57:39.034060 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.033830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:43583)
0812 18:57:39.033955 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.034004 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 18:57:39.034282 3639 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 18:57:39.156002 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:39.156373 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:39.156463 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:39.161139 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:39.161237 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:39.161334 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:57:39.167721 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:45421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=45049
--tserver_master_addrs=127.16.40.254:40205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:39.169073 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:39.170284 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:39.180835 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:39.180873 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:39.181178 3642 server_base.cc:1034] running on GCE node
W20260812 18:57:39.181275 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:39.181736 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:39.182525 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:39.183722 3642 hybrid_clock.cc:648] HybridClock initialized: now 1786561059183695 us; error 39 us; skew 500 ppm
Aug 12 18:57:39 dist-test-slave-rnrw krb5kdc[1947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561059, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:39.186779 3642 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:57:39.187785 3642 webserver.cc:460] Webserver started at http://127.16.40.195:45049/ using document root <none> and password file <none>
I20260812 18:57:39.188412 3642 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:39.188514 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:39.192902 3642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:57:39.195333 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:39.196749 3642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:39.196899 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "9b25ab2635074f14b03cfed6e359d403"
format_stamp: "Formatted at 2026-08-12 18:56:56 on dist-test-slave-rnrw"
server_key: "2e3b6c50930ef1716f776cadf59637b4"
server_key_iv: "239008b0b8ed8507714bb5a3bfc5e530"
server_key_version: "encryptionkey@0"
I20260812 18:57:39.197518 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:39.216774 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:39.220525 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:39.220862 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:39.221582 3642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:39.222587 3642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:39.222668 3642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:39.222744 3642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:39.222787 3642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:39.233137 3642 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:45421
I20260812 18:57:39.233180 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:45421 every 8 connection(s)
I20260812 18:57:39.234318 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:57:39.238864 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.236899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:48661)
0812 18:57:39.237888 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.238028 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":576,"threads_started":1}
W20260812 18:57:39.239463 3789 heartbeater.cc:644] Failed to heartbeat to 127.16.40.254:40205 (0 consecutive failures): Network error: Failed to ping master at 127.16.40.254:40205: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
I20260812 18:57:39.240219 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.239966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:40849)
0812 18:57:39.240099 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.240154 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 18:57:39.241048 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.240787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:56775)
0812 18:57:39.240922 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.240979 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 18:57:39.241045 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3642
W20260812 18:57:39.241299 3789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 18:57:39.852159 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:39.851634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:51519)
0812 18:57:39.851959 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:39.852030 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":79,"threads_started":1}
I20260812 18:57:40.035567 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.035102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:34641)
0812 18:57:40.035380 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:40.035449 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
I20260812 18:57:40.084203 3795 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:40.084409 3795 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:40.085656 3795 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
Aug 12 18:57:40 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.194@KRBTEST.COM
Aug 12 18:57:40 dist-test-slave-rnrw krb5kdc[1947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561058, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.195@KRBTEST.COM
I20260812 18:57:40.101188 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.085986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:45421 (local address 127.16.40.193:52277)
0812 18:57:40.086613 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:40.086631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:40.086744 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:40.088249 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:40.088256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:40.088848 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:40.089334 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:40.089349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.090832 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.090835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:40.091876 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:40.091888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.092778 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:40.092976 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:40.093838 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:40.093851 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:40.095752 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:40.098230 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:40.098233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:40.098235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:40.098587 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:40.098985 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:40.098990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:40.098995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:40.099187 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:40.100014 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:40.100017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:40.100295 (+ 278us) client_negotiation.cc:769] Sending connection context
0812 18:57:40.100502 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 18:57:40.100947 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":456,"spinlock_wait_cycles":57600,"thread_start_us":132,"threads_started":1}
I20260812 18:57:40.101238 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.086189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:52277 (local address 127.16.40.195:45421)
0812 18:57:40.086693 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:40.086705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:40.086745 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:57:40.087508 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:40.087514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:40.087845 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:40.088178 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:40.089540 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.090683 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092008 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.092647 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092800 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:40.095992 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:40.095997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:40.096009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:40.096045 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:40.098099 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:40.098718 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:40.098728 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:40.098736 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:40.098879 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:40.099317 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:40.099320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:40.099321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:40.099757 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:40.100040 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:40.100700 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 18:57:40.101033 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260812 18:57:40.101277 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.086186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:52513 (local address 127.16.40.194:33735)
0812 18:57:40.086639 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:40.086671 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:40.086720 (+ 49us) server_negotiation.cc:407] Connection header received
0812 18:57:40.087468 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:40.087475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:40.087844 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:40.088177 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:40.089523 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.090548 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092184 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.092724 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092892 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:40.095836 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:40.095845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:40.095865 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:40.095905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:40.097984 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:40.098758 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:40.098763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:40.098767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:40.098890 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:40.099278 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:40.099282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:40.099284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:40.099758 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:40.099907 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:40.100712 (+ 805us) server_negotiation.cc:299] Negotiation successful
0812 18:57:40.100999 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":119,"threads_started":1}
I20260812 18:57:40.101600 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.085986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33735 (local address 127.16.40.193:52513)
0812 18:57:40.086104 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:40.086122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:40.086232 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:40.088250 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:40.088256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:40.088847 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:40.089334 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:40.089348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.090706 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.090709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:40.091877 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:40.091888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:40.092855 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:40.092858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:40.092984 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:40.093717 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:40.093735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:40.095500 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:40.098132 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:40.098144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:40.098162 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:40.098604 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:40.099035 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:40.099038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:40.099039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:40.099173 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:40.099994 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:40.100002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:40.100296 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 18:57:40.100526 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 18:57:40.101041 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":457,"spinlock_wait_cycles":10496}
W20260812 18:57:40.104806 3346 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
W20260812 18:57:40.104991 3348 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
I20260812 18:57:40.105073 3348 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f20b5b385f946718d32186057ad5913; no voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9
I20260812 18:57:40.105320 3795 raft_consensus.cc:2726] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:57:40.242470 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.242014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:39981)
0812 18:57:40.242290 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:40.242353 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260812 18:57:40.853803 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:40.853267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:33061)
0812 18:57:40.853611 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:40.853685 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260812 18:57:41.036972 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:41.036413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:50497)
0812 18:57:41.036774 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:41.036839 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":84,"threads_started":1}
I20260812 18:57:41.244014 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:41.243500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:48333)
0812 18:57:41.243813 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:41.243882 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260812 18:57:41.735442 3806 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:41.735647 3806 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:41.736140 3806 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
W20260812 18:57:41.737346 3346 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
W20260812 18:57:41.737449 3348 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
I20260812 18:57:41.737495 3346 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f20b5b385f946718d32186057ad5913; no voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9
I20260812 18:57:41.737695 3806 raft_consensus.cc:2726] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:57:41.855177 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:41.854700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:56881)
0812 18:57:41.855007 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:41.855073 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260812 18:57:42.038400 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:42.037898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:60299)
0812 18:57:42.038223 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:42.038299 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260812 18:57:42.245440 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:42.244913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:48229)
0812 18:57:42.245256 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:42.245322 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260812 18:57:42.856658 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:42.856123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:35619)
0812 18:57:42.856416 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:42.856488 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260812 18:57:43.039808 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:43.039300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:54331)
0812 18:57:43.039643 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:43.039707 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":83,"threads_started":1}
I20260812 18:57:43.247008 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:43.246512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:33853)
0812 18:57:43.246824 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:43.246896 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260812 18:57:43.768949 3814 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:43.769116 3814 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:43.769701 3814 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
W20260812 18:57:43.770676 3348 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
W20260812 18:57:43.770889 3346 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
I20260812 18:57:43.771064 3346 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f20b5b385f946718d32186057ad5913; no voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9
I20260812 18:57:43.771250 3814 raft_consensus.cc:2726] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:57:43.858232 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:43.857699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:49789)
0812 18:57:43.858060 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:43.858128 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":100,"threads_started":1}
I20260812 18:57:44.041486 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:44.040982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:47847)
0812 18:57:44.041302 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:44.041376 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260812 18:57:44.248422 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:44.247990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:34219)
0812 18:57:44.248257 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:44.248316 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260812 18:57:44.859807 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:44.859234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:40035)
0812 18:57:44.859607 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:44.859697 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260812 18:57:45.042989 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:45.042479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:39953)
0812 18:57:45.042812 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:45.042882 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260812 18:57:45.249763 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:45.249291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:41417)
0812 18:57:45.249580 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:45.249648 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260812 18:57:45.861383 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:45.860839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:50005)
0812 18:57:45.861194 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:45.861271 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":97,"threads_started":1}
I20260812 18:57:46.044395 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:46.043849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:33261)
0812 18:57:46.044221 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:46.044289 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":82,"threads_started":1}
I20260812 18:57:46.251343 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:46.250784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:45517)
0812 18:57:46.251155 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:46.251229 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":81,"threads_started":1}
I20260812 18:57:46.622828 3829 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:46.623018 3829 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:46.623526 3829 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
W20260812 18:57:46.624430 3346 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
W20260812 18:57:46.624732 3348 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
I20260812 18:57:46.624836 3348 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f20b5b385f946718d32186057ad5913; no voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9
I20260812 18:57:46.625025 3829 raft_consensus.cc:2726] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:57:46.862884 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:46.862356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:56685)
0812 18:57:46.862718 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:46.862782 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":81,"threads_started":1}
I20260812 18:57:47.046026 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:47.045511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:36659)
0812 18:57:47.045845 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:47.045922 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
I20260812 18:57:47.252961 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:47.252405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:57531)
0812 18:57:47.252763 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:47.252834 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":88,"threads_started":1}
I20260812 18:57:47.864393 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:47.863871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:58765)
0812 18:57:47.864194 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:47.864258 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":112,"threads_started":1}
I20260812 18:57:48.047627 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:48.047050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:59529)
0812 18:57:48.047387 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:48.047469 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260812 18:57:48.254594 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:48.253974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:56061)
0812 18:57:48.254383 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:48.254455 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260812 18:57:48.866063 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:48.865578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:59259)
0812 18:57:48.865883 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:48.865956 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260812 18:57:49.049574 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:49.048878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:50373)
0812 18:57:49.049294 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:49.049368 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
I20260812 18:57:49.256203 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:49.255661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:48681)
0812 18:57:49.256014 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:49.256084 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260812 18:57:49.867710 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:49.867063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:37917)
0812 18:57:49.867496 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:49.867577 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":94,"threads_started":1}
I20260812 18:57:50.051010 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:50.050531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:46559)
0812 18:57:50.050833 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:50.050897 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260812 18:57:50.257876 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:50.257358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:36397)
0812 18:57:50.257697 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:50.257769 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260812 18:57:50.869297 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:50.868793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:52487)
0812 18:57:50.869117 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:50.869188 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260812 18:57:51.052673 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:51.052070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:47801)
0812 18:57:51.052388 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:51.052478 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260812 18:57:51.259327 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:51.258845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:38965)
0812 18:57:51.259155 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:51.259222 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20260812 18:57:51.832324 3846 raft_consensus.cc:482] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:51.832623 3846 raft_consensus.cc:504] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0ba1ee9905e4553b54853fe95dda7a9" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33735 } } peers { permanent_uuid: "9b25ab2635074f14b03cfed6e359d403" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 45421 } } peers { permanent_uuid: "4f20b5b385f946718d32186057ad5913" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 34591 } }
I20260812 18:57:51.833174 3846 leader_election.cc:288] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735), 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421)
W20260812 18:57:51.834214 3348 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0ba1ee9905e4553b54853fe95dda7a9 (127.16.40.194:33735): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
W20260812 18:57:51.834309 3346 leader_election.cc:341] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b25ab2635074f14b03cfed6e359d403 (127.16.40.195:45421): Not found: Tablet not found: e82e43d4b9984fae846332b16603194f
I20260812 18:57:51.834367 3348 leader_election.cc:302] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f20b5b385f946718d32186057ad5913; no voters: 9b25ab2635074f14b03cfed6e359d403, f0ba1ee9905e4553b54853fe95dda7a9
I20260812 18:57:51.834630 3846 raft_consensus.cc:2726] T e82e43d4b9984fae846332b16603194f P 4f20b5b385f946718d32186057ad5913 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 18:57:51.870865 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:51.870326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:44595)
0812 18:57:51.870695 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:51.870762 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260812 18:57:52.053975 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:52.053524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:39763)
0812 18:57:52.053811 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:52.053872 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260812 18:57:52.261299 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:52.260485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:44773)
0812 18:57:52.261054 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:52.261157 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260812 18:57:52.872426 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:52.871907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:38087)
0812 18:57:52.872254 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:52.872322 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260812 18:57:53.055639 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:53.055070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:55491)
0812 18:57:53.055373 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:53.055442 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
I20260812 18:57:53.262745 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:53.262230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:52591)
0812 18:57:53.262571 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:53.262641 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260812 18:57:53.874109 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:53.873570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.193:41599)
0812 18:57:53.873910 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:53.873991 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260812 18:57:54.057210 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:54.056739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.194:60595)
0812 18:57:54.057045 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:54.057109 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260812 18:57:54.252288 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3281
I20260812 18:57:54.263185 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3486
I20260812 18:57:54.265820 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:54.264921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:40205 (local address 127.16.40.195:53859)
0812 18:57:54.265659 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:54.265718 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.254:40205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":438,"threads_started":1}
I20260812 18:57:54.273079 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3642
2026-08-12T18:57:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59670 ms)
[----------] 30 tests from SecurityITest (262702 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:57:54 dist-test-slave-rnrw krb5kdc[3861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:57:54 dist-test-slave-rnrw krb5kdc[3861](info): set up 2 sockets
Aug 12 18:57:54 dist-test-slave-rnrw krb5kdc[3861](info): commencing operation
krb5kdc: starting...
W20260812 18:57:55.414745 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:57:55 dist-test-slave-rnrw krb5kdc[3861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:57:55.564450 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33083
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:55.687268 3874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:55.687628 3874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:55.687672 3874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:55.691934 3874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:57:55.692015 3874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:55.692035 3874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:55.692054 3874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:57:55.692070 3874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:57:55.697811 3874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:55.699026 3874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:55.700210 3874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:55.709846 3880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:55.709843 3882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:55.710005 3874 server_base.cc:1034] running on GCE node
W20260812 18:57:55.710214 3879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:55.710886 3874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:55.712205 3874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:55.713541 3874 hybrid_clock.cc:648] HybridClock initialized: now 1786561075713482 us; error 56 us; skew 500 ppm
Aug 12 18:57:55 dist-test-slave-rnrw krb5kdc[3861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561075, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:55.716889 3874 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:57:55.717914 3874 webserver.cc:460] Webserver started at http://127.16.40.254:34875/ using document root <none> and password file <none>
I20260812 18:57:55.718516 3874 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:55.718581 3874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:55.718760 3874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:55.721119 3874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "7b790ab28ef741168f924cbb881e7eac"
format_stamp: "Formatted at 2026-08-12 18:57:55 on dist-test-slave-rnrw"
server_key: "978c4787d0ea8b727ab79e6befecbdaf"
server_key_iv: "c43556faea47a281761ccff2b8c1fcbb"
server_key_version: "encryptionkey@0"
I20260812 18:57:55.721709 3874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "7b790ab28ef741168f924cbb881e7eac"
format_stamp: "Formatted at 2026-08-12 18:57:55 on dist-test-slave-rnrw"
server_key: "978c4787d0ea8b727ab79e6befecbdaf"
server_key_iv: "c43556faea47a281761ccff2b8c1fcbb"
server_key_version: "encryptionkey@0"
I20260812 18:57:55.726076 3874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:57:55.729025 3889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:55.730569 3874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:55.730780 3874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "7b790ab28ef741168f924cbb881e7eac"
format_stamp: "Formatted at 2026-08-12 18:57:55 on dist-test-slave-rnrw"
server_key: "978c4787d0ea8b727ab79e6befecbdaf"
server_key_iv: "c43556faea47a281761ccff2b8c1fcbb"
server_key_version: "encryptionkey@0"
I20260812 18:57:55.730929 3874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:55.748127 3874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:55.751910 3874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:55.752142 3874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:55.759975 3874 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33083
I20260812 18:57:55.759990 3947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33083 every 8 connection(s)
I20260812 18:57:55.761121 3874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:57:55.763309 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3874
I20260812 18:57:55.763481 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:57:55.764000 16547 external_mini_cluster.cc:1428] Setting key bda66dadfac0a158509db441c5c69785
I20260812 18:57:55.764456 3948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:55.772173 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: Bootstrap starting.
Aug 12 18:57:55 dist-test-slave-rnrw krb5kdc[3861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:55.775005 3948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:55.776072 3948 log.cc:824] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:55.778978 3948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: No bootstrap required, opened a new log
I20260812 18:57:55.781453 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:55.765585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.16.40.254:33083)
0812 18:57:55.766063 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:55.766080 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:55.766122 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:57:55.766797 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:55.766815 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:55.767166 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:55.767474 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:55.768444 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:55.769581 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:55.770801 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:55.771330 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:55.771456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:55.774380 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:55.774386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:55.774402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:55.774437 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:55.776857 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:55.777521 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:55.777528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:55.777536 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:55.777740 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:55.778161 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:55.778166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:55.778171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:55.778808 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:55.779035 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:55.779624 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 18:57:55.780129 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":157,"threads_started":1}
I20260812 18:57:55.783793 3948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } }
I20260812 18:57:55.784112 3948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:55.784174 3948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b790ab28ef741168f924cbb881e7eac, State: Initialized, Role: FOLLOWER
I20260812 18:57:55.784755 3948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } }
I20260812 18:57:55.784948 3948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:57:55.785004 3948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:57:55.785135 3948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:55.786767 3948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } }
I20260812 18:57:55.787256 3948 leader_election.cc:302] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b790ab28ef741168f924cbb881e7eac; no voters:
I20260812 18:57:55.788101 3948 leader_election.cc:288] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:57:55.788292 3956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:55.788789 3956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 1 LEADER]: Becoming Leader. State: Replica: 7b790ab28ef741168f924cbb881e7eac, State: Running, Role: LEADER
I20260812 18:57:55.789429 3956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } }
I20260812 18:57:55.790130 3948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:57:55.792395 3954 raft_consensus.cc:465] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [term 1 LEADER]: Not starting pre-election -- already a leader
I20260812 18:57:55.793560 3957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b790ab28ef741168f924cbb881e7eac. Latest consensus state: current_term: 1 leader_uuid: "7b790ab28ef741168f924cbb881e7eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } } }
I20260812 18:57:55.793731 3957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:55.794060 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b790ab28ef741168f924cbb881e7eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b790ab28ef741168f924cbb881e7eac" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33083 } } }
I20260812 18:57:55.794167 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac [sys.catalog]: This master's current role is: LEADER
W20260812 18:57:55.796746 3973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:57:55.796855 3973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:57:55.796938 3974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:57:55.797712 3974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:57:55.805527 3974 catalog_manager.cc:1347] Generated new cluster ID: 8e696588850e4e729c0eb44c8c6924c1
I20260812 18:57:55.805619 3974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:57:55.821789 3974 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:57:55.823192 3974 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:57:55.834468 3974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: Generated new TSK 0
I20260812 18:57:55.835548 3974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:57:55.868902 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:55.996193 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:55.996512 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:55.996634 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:56.001195 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:56.001286 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:56.001371 3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:57:56.007436 3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:56.008772 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:56.009944 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:56.021371 3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:56.021564 3981 server_base.cc:1034] running on GCE node
W20260812 18:57:56.021387 3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:56.021389 3987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:56.022534 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:56.023283 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:56.024617 3981 hybrid_clock.cc:648] HybridClock initialized: now 1786561076024527 us; error 34 us; skew 500 ppm
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:56.028069 3981 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:57:56.029251 3981 webserver.cc:460] Webserver started at http://127.16.40.193:43163/ using document root <none> and password file <none>
I20260812 18:57:56.029857 3981 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:56.029924 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:56.030128 3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:56.032315 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "d42fec87f52645c5bf0471a09149356c"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "c70a2ac69ea54350f6747943e7562a3e"
server_key_iv: "f38317b21acc9b1b9c4379aa38e38d75"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.033185 3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "d42fec87f52645c5bf0471a09149356c"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "c70a2ac69ea54350f6747943e7562a3e"
server_key_iv: "f38317b21acc9b1b9c4379aa38e38d75"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.037609 3981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:57:56.040335 3996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.041889 3981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:57:56.042069 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "d42fec87f52645c5bf0471a09149356c"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "c70a2ac69ea54350f6747943e7562a3e"
server_key_iv: "f38317b21acc9b1b9c4379aa38e38d75"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.042205 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:56.080598 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:56.084260 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:56.084509 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:56.085222 3981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:56.086194 3981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:56.086244 3981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.086318 3981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:56.086349 3981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.097255 3981 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:44515
I20260812 18:57:56.097290 4123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:44515 every 8 connection(s)
I20260812 18:57:56.098443 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:57:56.108210 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 3981
I20260812 18:57:56.108412 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:57:56.108801 16547 external_mini_cluster.cc:1428] Setting key ed2000ecb48f697adc5e5369cd7c0014
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:56.116097 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.100774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:42393 (local address 127.16.40.254:33083)
0812 18:57:56.100927 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.100932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.102090 (+ 1158us) server_negotiation.cc:407] Connection header received
0812 18:57:56.103695 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.103699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.103764 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.103858 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:56.105332 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.105947 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.107629 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.107991 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.108132 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.111363 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:56.111367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:56.111369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:56.111398 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:56.113340 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.114014 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.114021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.114024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.114140 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.114517 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.114523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.114527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.114825 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:56.114950 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.115647 (+ 697us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.115838 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:57:56.117057 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.101229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33083 (local address 127.16.40.193:42393)
0812 18:57:56.101885 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.101936 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.103269 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.104076 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.104084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.104579 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:56.105147 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.105163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.106163 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.106168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.107305 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.107318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.108195 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.108199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.108408 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.109241 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:56.109260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:56.111147 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:56.113494 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.113501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.113519 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.113879 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.114239 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.114242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.114243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.114405 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.114989 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:56.114996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:56.115292 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.115531 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.115864 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":164,"threads_started":1}
I20260812 18:57:56.118688 4124 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33083
I20260812 18:57:56.119022 4124 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:56.119630 4124 heartbeater.cc:511] Master 127.16.40.254:33083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
I20260812 18:57:56.121542 3907 ts_manager.cc:194] Registered new tserver with Master: d42fec87f52645c5bf0471a09149356c (127.16.40.193:44515)
I20260812 18:57:56.122721 3907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:42393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:57:56.145020 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:56.267877 4132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:56.268210 4132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:56.268258 4132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:56.272675 4132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:56.272756 4132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:56.272821 4132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:57:56.278658 4132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:56.279911 4132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:56.281054 4132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:56.291607 4139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:56.291631 4141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:56.291664 4132 server_base.cc:1034] running on GCE node
W20260812 18:57:56.291605 4138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:56.292497 4132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:56.293349 4132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:56.294576 4132 hybrid_clock.cc:648] HybridClock initialized: now 1786561076294558 us; error 32 us; skew 500 ppm
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:56.297636 4132 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:57:56.298733 4132 webserver.cc:460] Webserver started at http://127.16.40.194:44621/ using document root <none> and password file <none>
I20260812 18:57:56.299397 4132 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:56.299466 4132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:56.299714 4132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:56.302049 4132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "405ec231da9b4822b006b9a35876eb2b"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "5c77fee6bb63b68411d8bb676d0a9fcc"
server_key_iv: "8dee61ddef7bebc460d450d8d0eb4e84"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.302732 4132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "405ec231da9b4822b006b9a35876eb2b"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "5c77fee6bb63b68411d8bb676d0a9fcc"
server_key_iv: "8dee61ddef7bebc460d450d8d0eb4e84"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.307094 4132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:57:56.309989 4149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.311338 4132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:57:56.311493 4132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "405ec231da9b4822b006b9a35876eb2b"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "5c77fee6bb63b68411d8bb676d0a9fcc"
server_key_iv: "8dee61ddef7bebc460d450d8d0eb4e84"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.311630 4132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:56.335242 4132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:56.338865 4132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:56.339103 4132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:56.339788 4132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:56.340747 4132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:56.340796 4132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.340867 4132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:56.340901 4132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.351199 4132 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:46253
I20260812 18:57:56.351470 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:46253 every 8 connection(s)
I20260812 18:57:56.352360 4132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:57:56.353679 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4132
I20260812 18:57:56.353780 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:57:56.354120 16547 external_mini_cluster.cc:1428] Setting key 765dd4cc91499cae3bf2914d4720b5e6
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:57:56.369462 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.354848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:34277 (local address 127.16.40.254:33083)
0812 18:57:56.355018 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.355022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.356226 (+ 1204us) server_negotiation.cc:407] Connection header received
0812 18:57:56.357838 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.357841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.357907 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.357989 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:56.359427 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.359981 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.361430 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.361696 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.361829 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.364931 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:56.364935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:56.364937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:56.364969 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:56.366703 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.367472 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.367480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.367482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.367575 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.367994 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.367998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.368000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.368270 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:56.368425 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.369133 (+ 708us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.369288 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 18:57:56.370427 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.355388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33083 (local address 127.16.40.194:34277)
0812 18:57:56.356053 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.356116 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.357553 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.358213 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.358221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.358632 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:56.359219 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.359235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.360149 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.360152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.361269 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.361282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.361893 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.361897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.362119 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.362894 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:56.362914 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:56.364688 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:56.366944 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.366953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.366983 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.367321 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.367688 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.367692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.367694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.367879 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.368424 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:56.368434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:56.368855 (+ 421us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.369071 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.369460 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":190,"threads_started":1}
I20260812 18:57:56.371908 4285 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33083
I20260812 18:57:56.372288 4285 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:56.373051 4285 heartbeater.cc:511] Master 127.16.40.254:33083 requested a full tablet report, sending...
I20260812 18:57:56.374473 3907 ts_manager.cc:194] Registered new tserver with Master: 405ec231da9b4822b006b9a35876eb2b (127.16.40.194:46253)
I20260812 18:57:56.375485 3907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:34277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:57:56.387879 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:56.512741 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:56.513075 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:56.513123 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:56.517681 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:56.517767 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:56.517834 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:57:56.523635 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:56.524914 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:56.526007 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:56.536818 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:56.536823 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:56.536818 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:56.537163 4292 server_base.cc:1034] running on GCE node
I20260812 18:57:56.537863 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:56.538632 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:56.539827 4292 hybrid_clock.cc:648] HybridClock initialized: now 1786561076539785 us; error 53 us; skew 500 ppm
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:56.543252 4292 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:57:56.544407 4292 webserver.cc:460] Webserver started at http://127.16.40.195:34297/ using document root <none> and password file <none>
I20260812 18:57:56.545182 4292 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:56.545288 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:56.545564 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:56.548035 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "a1604b706afa4da98b60293dcad29204"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "a533a8f02d3950f5969c5c72f30ad96f"
server_key_iv: "82bc693eacc3918f2d8177b6d49179a1"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.548723 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "a1604b706afa4da98b60293dcad29204"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "a533a8f02d3950f5969c5c72f30ad96f"
server_key_iv: "82bc693eacc3918f2d8177b6d49179a1"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.553038 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:57:56.555910 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.557626 4292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:56.557754 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "a1604b706afa4da98b60293dcad29204"
format_stamp: "Formatted at 2026-08-12 18:57:56 on dist-test-slave-rnrw"
server_key: "a533a8f02d3950f5969c5c72f30ad96f"
server_key_iv: "82bc693eacc3918f2d8177b6d49179a1"
server_key_version: "encryptionkey@0"
I20260812 18:57:56.557864 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:56.574180 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:56.577808 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:56.577992 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:56.578619 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:56.579564 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:56.579617 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.579658 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:56.579674 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:56.589545 4292 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:37349
I20260812 18:57:56.589572 4445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:37349 every 8 connection(s)
I20260812 18:57:56.590610 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:57:56.596940 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4292
I20260812 18:57:56.597111 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:57:56.597462 16547 external_mini_cluster.cc:1428] Setting key 8f1982da07137adfbcb67658d920f345
Aug 12 18:57:56 dist-test-slave-rnrw krb5kdc[3861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561076, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:56.607432 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.592733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:32923 (local address 127.16.40.254:33083)
0812 18:57:56.592882 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.592886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.593807 (+ 921us) server_negotiation.cc:407] Connection header received
0812 18:57:56.595210 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.595214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.595297 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.595383 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:56.596956 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.597888 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.599897 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.600232 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.600330 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.603250 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:56.603253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:56.603256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:56.603284 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:56.605134 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.605750 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.605753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.605754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.605841 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.606204 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.606206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.606207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.606473 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:56.606557 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.607162 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.607283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:57:56.608424 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.593160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33083 (local address 127.16.40.195:32923)
0812 18:57:56.593636 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.593681 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.594960 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.595619 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.595627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.596054 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:56.596741 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.596761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.598066 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.598074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.599655 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.599668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.600398 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.600402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.600656 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.601298 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:57:56.601317 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:56.603050 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:56.605265 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.605272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.605288 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.605603 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.605935 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:57:56.605938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:57:56.605940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:57:56.606102 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:57:56.606586 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:56.606592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:56.606883 (+ 291us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.607073 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.607392 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260812 18:57:56.609831 4446 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33083
I20260812 18:57:56.610141 4446 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:56.610785 4446 heartbeater.cc:511] Master 127.16.40.254:33083 requested a full tablet report, sending...
I20260812 18:57:56.611971 3906 ts_manager.cc:194] Registered new tserver with Master: a1604b706afa4da98b60293dcad29204 (127.16.40.195:37349)
I20260812 18:57:56.612694 3906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:32923
I20260812 18:57:56.624320 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:57:56.634035 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.626567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.16.40.254:33083)
0812 18:57:56.626731 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.626737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.626969 (+ 232us) server_negotiation.cc:407] Connection header received
0812 18:57:56.627119 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.627121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.627168 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.627235 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:56.627958 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.628493 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.629362 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.629569 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.629658 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.631048 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:56.631058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:56.631060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:56.631096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:56.632391 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.632858 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.632861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.632862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.632941 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:56.633179 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:56.633181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:56.633182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:56.633408 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:56.633511 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.633733 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.633836 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:57:56.637363 3906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:57:56.640537 3906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:57:56.656754 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.650975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.16.40.195:37349)
0812 18:57:56.651477 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.651491 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.651514 (+ 23us) server_negotiation.cc:407] Connection header received
0812 18:57:56.651590 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.651596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.651766 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.651938 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:56.652461 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.653568 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.655158 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.655924 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.656036 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.656189 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.656434 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.656579 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":180,"threads_started":1}
I20260812 18:57:56.656765 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.649885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:37349 (local address 127.0.0.1:55144)
0812 18:57:56.650610 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.650664 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.650799 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.652146 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.652149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.652170 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:56.652327 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.652335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.653751 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.653754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.655011 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.655021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.656066 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.656070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.656204 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.656252 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.656386 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.656532 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":89,"thread_start_us":97,"threads_started":1}
I20260812 18:57:56.656952 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.650183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44515 (local address 127.0.0.1:38738)
0812 18:57:56.651167 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.651183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.651303 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.651882 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.651892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.651917 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:56.652062 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.652069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.653684 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.653688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.655407 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.655427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.656539 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.656543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.656655 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.656679 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.656732 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.656812 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":86,"thread_start_us":132,"threads_started":1}
I20260812 18:57:56.657768 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.650344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.16.40.193:44515)
0812 18:57:56.650837 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.650846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.651275 (+ 429us) server_negotiation.cc:407] Connection header received
0812 18:57:56.651427 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.651434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.651679 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.651854 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:56.652219 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.653517 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.655565 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.656398 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.657138 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.657430 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.657481 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.657580 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":133,"threads_started":1}
I20260812 18:57:56.658249 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.650650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:46253 (local address 127.0.0.1:48340)
0812 18:57:56.652156 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.652171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.652262 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.652803 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.652807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.652827 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:56.652936 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.652945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.654297 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.654304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.655568 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.655576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.657649 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.657653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.657756 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.657789 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.658034 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.658114 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260812 18:57:56.658874 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.650881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.16.40.194:46253)
0812 18:57:56.651712 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.651737 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.652260 (+ 523us) server_negotiation.cc:407] Connection header received
0812 18:57:56.652375 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.652384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.652656 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.652823 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:56.653047 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.654129 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.655699 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.656609 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.656717 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.658574 (+ 1857us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.658632 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.658731 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":145,"threads_started":1}
I20260812 18:57:56.660308 4050 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c (DEFAULT_TABLE table=test-table [id=1e6ea6c9509c447aac2a6b2131241381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:56.660305 4368 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c (DEFAULT_TABLE table=test-table [id=1e6ea6c9509c447aac2a6b2131241381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:56.661396 4368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2424abe6734bb19f4e23013bbfb60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:56.661437 4050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2424abe6734bb19f4e23013bbfb60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:56.661629 4202 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c (DEFAULT_TABLE table=test-table [id=1e6ea6c9509c447aac2a6b2131241381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:56.662585 4202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a2424abe6734bb19f4e23013bbfb60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:56.668506 4464 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Bootstrap starting.
I20260812 18:57:56.670058 4465 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Bootstrap starting.
I20260812 18:57:56.671350 4464 tablet_bootstrap.cc:654] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:56.671995 4465 tablet_bootstrap.cc:654] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:56.672732 4464 log.cc:824] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:56.672780 4465 log.cc:824] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:56.672935 4466 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Bootstrap starting.
I20260812 18:57:56.674855 4464 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: No bootstrap required, opened a new log
I20260812 18:57:56.674983 4465 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: No bootstrap required, opened a new log
I20260812 18:57:56.675146 4464 ts_tablet_manager.cc:1397] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 18:57:56.675160 4466 tablet_bootstrap.cc:654] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:56.675196 4465 ts_tablet_manager.cc:1397] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 18:57:56.676506 4466 log.cc:824] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:56.679040 4466 tablet_bootstrap.cc:492] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: No bootstrap required, opened a new log
I20260812 18:57:56.679272 4466 ts_tablet_manager.cc:1397] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 18:57:56.679126 4465 raft_consensus.cc:348] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.679349 4465 raft_consensus.cc:374] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:56.679383 4465 raft_consensus.cc:729] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42fec87f52645c5bf0471a09149356c, State: Initialized, Role: FOLLOWER
I20260812 18:57:56.679520 4464 raft_consensus.cc:348] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.679806 4464 raft_consensus.cc:374] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:56.679864 4464 raft_consensus.cc:729] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1604b706afa4da98b60293dcad29204, State: Initialized, Role: FOLLOWER
I20260812 18:57:56.679854 4465 consensus_queue.cc:260] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.680361 4464 consensus_queue.cc:260] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.680742 4124 heartbeater.cc:503] Master 127.16.40.254:33083 was elected leader, sending a full tablet report...
I20260812 18:57:56.681036 4465 ts_tablet_manager.cc:1428] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260812 18:57:56.681424 4464 ts_tablet_manager.cc:1428] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260812 18:57:56.681602 4446 heartbeater.cc:503] Master 127.16.40.254:33083 was elected leader, sending a full tablet report...
I20260812 18:57:56.683776 4466 raft_consensus.cc:348] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.684111 4466 raft_consensus.cc:374] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:56.684190 4466 raft_consensus.cc:729] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 405ec231da9b4822b006b9a35876eb2b, State: Initialized, Role: FOLLOWER
I20260812 18:57:56.684725 4466 consensus_queue.cc:260] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.685546 4285 heartbeater.cc:503] Master 127.16.40.254:33083 was elected leader, sending a full tablet report...
I20260812 18:57:56.686000 4466 ts_tablet_manager.cc:1428] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260812 18:57:56.841848 4448 tablet.cc:2369] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:57:56.850194 4125 tablet.cc:2369] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:57:56.853713 4286 tablet.cc:2369] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:57:56.903486 4472 raft_consensus.cc:482] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:56.903713 4472 raft_consensus.cc:504] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.904917 4472 leader_election.cc:288] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d42fec87f52645c5bf0471a09149356c (127.16.40.193:44515), 405ec231da9b4822b006b9a35876eb2b (127.16.40.194:46253)
I20260812 18:57:56.910013 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.905421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:56717 (local address 127.16.40.194:46253)
0812 18:57:56.905595 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.905599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.906232 (+ 633us) server_negotiation.cc:407] Connection header received
0812 18:57:56.906330 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.906333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.906391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.906486 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:56.906805 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.907415 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.908844 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.909428 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.909549 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.909595 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.909844 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.909914 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:57:56.909996 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.905421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:48357 (local address 127.16.40.193:44515)
0812 18:57:56.905664 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.905669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.905685 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:57:56.905757 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.905762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.905828 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.905914 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:56.906332 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.907078 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.908686 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.909316 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.909446 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.909525 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.909761 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.909866 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:57:56.909992 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.905256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44515 (local address 127.16.40.195:48357)
0812 18:57:56.905376 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.905394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.905542 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.905976 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.905980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.906009 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:56.906181 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.906193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.907205 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.907208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.908515 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.908526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.909441 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.909443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.909571 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.909651 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.909732 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.909822 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":24}
I20260812 18:57:56.909999 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.905256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:46253 (local address 127.16.40.195:56717)
0812 18:57:56.906105 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:56.906123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:56.906246 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:56.906556 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:56.906558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:56.906577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:56.906687 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.906692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.907602 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.907606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.908740 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:56.908747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.909584 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.909587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:56.909730 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.909751 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 18:57:56.909815 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 18:57:56.909861 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":397,"thread_start_us":150,"threads_started":1}
I20260812 18:57:56.910816 4072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2424abe6734bb19f4e23013bbfb60c" candidate_uuid: "a1604b706afa4da98b60293dcad29204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fec87f52645c5bf0471a09149356c" is_pre_election: true
I20260812 18:57:56.910822 4228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2424abe6734bb19f4e23013bbfb60c" candidate_uuid: "a1604b706afa4da98b60293dcad29204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "405ec231da9b4822b006b9a35876eb2b" is_pre_election: true
I20260812 18:57:56.911158 4072 raft_consensus.cc:2454] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1604b706afa4da98b60293dcad29204 in term 0.
I20260812 18:57:56.911168 4228 raft_consensus.cc:2454] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1604b706afa4da98b60293dcad29204 in term 0.
I20260812 18:57:56.911752 4312 leader_election.cc:302] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 405ec231da9b4822b006b9a35876eb2b, a1604b706afa4da98b60293dcad29204, d42fec87f52645c5bf0471a09149356c; no voters:
I20260812 18:57:56.912098 4472 raft_consensus.cc:2781] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:57:56.912189 4472 raft_consensus.cc:482] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:57:56.912242 4472 raft_consensus.cc:3037] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:56.913419 4472 raft_consensus.cc:504] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.913894 4472 leader_election.cc:288] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [CANDIDATE]: Term 1 election: Requested vote from peers d42fec87f52645c5bf0471a09149356c (127.16.40.193:44515), 405ec231da9b4822b006b9a35876eb2b (127.16.40.194:46253)
I20260812 18:57:56.914323 4072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2424abe6734bb19f4e23013bbfb60c" candidate_uuid: "a1604b706afa4da98b60293dcad29204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d42fec87f52645c5bf0471a09149356c"
I20260812 18:57:56.914453 4228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a2424abe6734bb19f4e23013bbfb60c" candidate_uuid: "a1604b706afa4da98b60293dcad29204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "405ec231da9b4822b006b9a35876eb2b"
I20260812 18:57:56.914484 4072 raft_consensus.cc:3037] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:56.914592 4228 raft_consensus.cc:3037] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:56.915966 4072 raft_consensus.cc:2454] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1604b706afa4da98b60293dcad29204 in term 1.
I20260812 18:57:56.915961 4228 raft_consensus.cc:2454] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1604b706afa4da98b60293dcad29204 in term 1.
I20260812 18:57:56.916383 4313 leader_election.cc:302] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1604b706afa4da98b60293dcad29204, d42fec87f52645c5bf0471a09149356c; no voters:
I20260812 18:57:56.916612 4472 raft_consensus.cc:2781] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:56.916848 4472 raft_consensus.cc:686] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 1 LEADER]: Becoming Leader. State: Replica: a1604b706afa4da98b60293dcad29204, State: Running, Role: LEADER
I20260812 18:57:56.917210 4472 consensus_queue.cc:237] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } }
I20260812 18:57:56.920890 3907 catalog_manager.cc:5495] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1604b706afa4da98b60293dcad29204 (127.16.40.195). New cstate: current_term: 1 leader_uuid: "a1604b706afa4da98b60293dcad29204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1604b706afa4da98b60293dcad29204" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 37349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 } health_report { overall_health: UNKNOWN } } }
I20260812 18:57:56.957402 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:56.953555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.16.40.195:37349)
0812 18:57:56.953714 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:56.953721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:56.953737 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:57:56.953830 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:56.953832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:56.953901 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:56.954027 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:57:56.954324 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.954885 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.955788 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:56.956010 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:56.956122 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:56.956467 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:57:56.956970 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:57:56.957110 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:56.957156 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 18:57:56.957223 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:57:56.963987 4072 raft_consensus.cc:1261] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 1 FOLLOWER]: Refusing update from remote peer a1604b706afa4da98b60293dcad29204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:56.963989 4228 raft_consensus.cc:1261] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 1 FOLLOWER]: Refusing update from remote peer a1604b706afa4da98b60293dcad29204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:56.964836 4472 consensus_queue.cc:1035] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d42fec87f52645c5bf0471a09149356c" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:56.965109 4475 consensus_queue.cc:1035] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "405ec231da9b4822b006b9a35876eb2b" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:56.974344 4481 mvcc.cc:205] Tried to move back new op lower bound from 7317754171236012032 to 7317754171055480832. Current Snapshot: MvccSnapshot[applied={T|T < 7317754171236012032}]
I20260812 18:57:56.985227 3907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:56.985455 3907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:56.988411 3907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b790ab28ef741168f924cbb881e7eac: Sending DeleteTablet for 3 replicas of tablet 2a2424abe6734bb19f4e23013bbfb60c
I20260812 18:57:56.989404 4368 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:56 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:55144
I20260812 18:57:56.989401 4050 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:56 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:38738
I20260812 18:57:56.989563 4202 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a2424abe6734bb19f4e23013bbfb60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:56 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:48340
I20260812 18:57:56.990114 4491 tablet_replica.cc:330] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c: stopping tablet replica
I20260812 18:57:56.990314 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3981
I20260812 18:57:56.990322 4493 tablet_replica.cc:330] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: stopping tablet replica
I20260812 18:57:56.990535 4491 raft_consensus.cc:2229] T 2a2424abe6734bb19f4e23013bbfb60c P d42fec87f52645c5bf0471a09149356c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:56.990747 4493 raft_consensus.cc:2229] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:57:56.991080 4494 tablet_replica.cc:330] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: stopping tablet replica
I20260812 18:57:56.991376 4493 raft_consensus.cc:2258] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:56.991380 4494 raft_consensus.cc:2229] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:56.991716 4494 raft_consensus.cc:2258] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:56.993461 4494 ts_tablet_manager.cc:1905] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:56.993461 4493 ts_tablet_manager.cc:1905] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:56.996430 4494 ts_tablet_manager.cc:1918] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:56.996430 4493 ts_tablet_manager.cc:1918] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:56.996587 4494 log.cc:1196] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/2a2424abe6734bb19f4e23013bbfb60c
I20260812 18:57:56.996742 4493 log.cc:1196] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/2a2424abe6734bb19f4e23013bbfb60c
I20260812 18:57:56.996943 4494 ts_tablet_manager.cc:1939] T 2a2424abe6734bb19f4e23013bbfb60c P 405ec231da9b4822b006b9a35876eb2b: Deleting consensus metadata
I20260812 18:57:56.997248 4493 ts_tablet_manager.cc:1939] T 2a2424abe6734bb19f4e23013bbfb60c P a1604b706afa4da98b60293dcad29204: Deleting consensus metadata
I20260812 18:57:56.998448 3892 catalog_manager.cc:4841] TS 405ec231da9b4822b006b9a35876eb2b (127.16.40.194:46253): tablet 2a2424abe6734bb19f4e23013bbfb60c (table test-table [id=1e6ea6c9509c447aac2a6b2131241381]) successfully deleted
I20260812 18:57:56.998711 3892 catalog_manager.cc:4841] TS a1604b706afa4da98b60293dcad29204 (127.16.40.195:37349): tablet 2a2424abe6734bb19f4e23013bbfb60c (table test-table [id=1e6ea6c9509c447aac2a6b2131241381]) successfully deleted
W20260812 18:57:57.004652 3893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:44515 (error 108)
I20260812 18:57:57.005357 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4132
I20260812 18:57:57.009961 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.009489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:44515 (local address 127.0.0.1:38744)
0812 18:57:57.009751 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.009849 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260812 18:57:57.010380 3893 catalog_manager.cc:4584] TS d42fec87f52645c5bf0471a09149356c (127.16.40.193:44515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a2424abe6734bb19f4e23013bbfb60c: Network error: Client connection negotiation failed: client connection to 127.16.40.193:44515: connect: Connection refused (error 111)
I20260812 18:57:57.016108 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4292
I20260812 18:57:57.027727 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 3874
2026-08-12T18:57:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2753 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T18:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:57:57Z Disabled control of system clock
I20260812 18:57:57.064471 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33015
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:57:57.186866 4499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:57.187215 4499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:57.187258 4499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:57.191645 4499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:57:57.191735 4499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:57.191751 4499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:57.191768 4499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:57:57.191784 4499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:57:57.197189 4499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:57.198557 4499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:57.200165 4499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:57.209700 4507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.209649 4509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.209641 4506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:57.210110 4499 server_base.cc:1034] running on GCE node
I20260812 18:57:57.210762 4499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:57.212057 4499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:57.213299 4499 hybrid_clock.cc:648] HybridClock initialized: now 1786561077213235 us; error 84 us; skew 500 ppm
I20260812 18:57:57.215653 4499 webserver.cc:460] Webserver started at http://127.16.40.254:43095/ using document root <none> and password file <none>
I20260812 18:57:57.216365 4499 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:57.216596 4499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:57.216877 4499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:57.219210 4499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "c67fb1515cc7444bb0b5668f28ca896b"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "d572122616c3491060c39ed7464e7eea"
server_key_iv: "e1c1ade6fd3bb6ceb576256e4f4ad87f"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.219887 4499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "c67fb1515cc7444bb0b5668f28ca896b"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "d572122616c3491060c39ed7464e7eea"
server_key_iv: "e1c1ade6fd3bb6ceb576256e4f4ad87f"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.224218 4499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 18:57:57.227480 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.229139 4499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:57.229333 4499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "c67fb1515cc7444bb0b5668f28ca896b"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "d572122616c3491060c39ed7464e7eea"
server_key_iv: "e1c1ade6fd3bb6ceb576256e4f4ad87f"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.229561 4499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:57.248739 4499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:57.249657 4499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:57.249910 4499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:57.258286 4499 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33015
I20260812 18:57:57.258301 4589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33015 every 8 connection(s)
I20260812 18:57:57.259371 4499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:57:57.262571 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4499
I20260812 18:57:57.262744 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:57:57.262893 4590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:57.263130 16547 external_mini_cluster.cc:1428] Setting key ff58380c3ce9633a4ae9b4fd6c6454c0
I20260812 18:57:57.269933 4590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: Bootstrap starting.
I20260812 18:57:57.272333 4590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:57.273190 4590 log.cc:824] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:57.273300 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.264798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.16.40.254:33015)
0812 18:57:57.265592 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.265609 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.265652 (+ 43us) server_negotiation.cc:407] Connection header received
0812 18:57:57.266368 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.266386 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.266816 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.267235 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:57.267718 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.268775 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.269684 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.270227 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.270407 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.271364 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:57.271369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:57.271383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:57:57.271782 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:57.271843 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:57:57.271865 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:57.271981 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.272106 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.272387 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":169,"threads_started":1}
I20260812 18:57:57.275264 4590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: No bootstrap required, opened a new log
I20260812 18:57:57.279197 4590 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } }
I20260812 18:57:57.279500 4590 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:57.279594 4590 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67fb1515cc7444bb0b5668f28ca896b, State: Initialized, Role: FOLLOWER
I20260812 18:57:57.280107 4590 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } }
I20260812 18:57:57.280269 4590 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:57:57.280364 4590 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:57:57.280472 4590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:57.281522 4590 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } }
I20260812 18:57:57.281901 4590 leader_election.cc:302] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67fb1515cc7444bb0b5668f28ca896b; no voters:
I20260812 18:57:57.282298 4590 leader_election.cc:288] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:57:57.282387 4597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:57.282603 4597 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [term 1 LEADER]: Becoming Leader. State: Replica: c67fb1515cc7444bb0b5668f28ca896b, State: Running, Role: LEADER
I20260812 18:57:57.282917 4597 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } }
I20260812 18:57:57.283751 4590 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:57:57.284780 4599 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67fb1515cc7444bb0b5668f28ca896b. Latest consensus state: current_term: 1 leader_uuid: "c67fb1515cc7444bb0b5668f28ca896b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } } }
I20260812 18:57:57.284925 4599 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:57.285264 4598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67fb1515cc7444bb0b5668f28ca896b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67fb1515cc7444bb0b5668f28ca896b" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33015 } } }
I20260812 18:57:57.285359 4598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:57.285537 4606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:57:57.289120 4606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:57:57.295107 4606 catalog_manager.cc:1347] Generated new cluster ID: c593326cd1d148078aed659c01c31973
I20260812 18:57:57.295200 4606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:57:57.311362 4606 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:57:57.312853 4606 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:57:57.323030 4606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: Generated new TSK 0
I20260812 18:57:57.323972 4606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:57:57.329886 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--builtin_ntp_servers=127.16.40.212:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:57:57.458128 4618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:57.458431 4618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:57.458473 4618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:57.462713 4618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:57.462786 4618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:57.462844 4618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:57:57.467886 4618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:57.469129 4618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:57.470646 4618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:57.480999 4624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.481015 4623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.481050 4626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:57.481019 4618 server_base.cc:1034] running on GCE node
I20260812 18:57:57.481775 4618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:57.482558 4618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:57.483770 4618 hybrid_clock.cc:648] HybridClock initialized: now 1786561077483755 us; error 29 us; skew 500 ppm
I20260812 18:57:57.485971 4618 webserver.cc:460] Webserver started at http://127.16.40.193:34145/ using document root <none> and password file <none>
I20260812 18:57:57.486614 4618 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:57.486724 4618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:57.486982 4618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:57.489275 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "e6dceddb74364a9999d52912057df18d"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "36ee1d4cc6429d9f61d9084c54a5f2ee"
server_key_iv: "9f716736744e1accdbd0fe8ec8494f33"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.489910 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "e6dceddb74364a9999d52912057df18d"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "36ee1d4cc6429d9f61d9084c54a5f2ee"
server_key_iv: "9f716736744e1accdbd0fe8ec8494f33"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.494123 4618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:57:57.496965 4634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.498518 4618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:57:57.498643 4618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "e6dceddb74364a9999d52912057df18d"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "36ee1d4cc6429d9f61d9084c54a5f2ee"
server_key_iv: "9f716736744e1accdbd0fe8ec8494f33"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.498760 4618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:57.517932 4618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:57.518671 4618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:57.518839 4618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:57.519426 4618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:57.520419 4618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:57.520469 4618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.520519 4618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:57.520535 4618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.530224 4618 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:36627
I20260812 18:57:57.530289 4772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:36627 every 8 connection(s)
I20260812 18:57:57.531270 4618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:57:57.538607 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4618
I20260812 18:57:57.538726 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:57:57.539070 16547 external_mini_cluster.cc:1428] Setting key 1cc43766ec68b7b54bf322667e8fd8c4
I20260812 18:57:57.541366 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--builtin_ntp_servers=127.16.40.212:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:57:57.542395 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.533578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:59061 (local address 127.16.40.254:33015)
0812 18:57:57.533728 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.533732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.535057 (+ 1325us) server_negotiation.cc:407] Connection header received
0812 18:57:57.536641 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.536645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.536716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.536827 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:57.537820 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.538369 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.539856 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.540098 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.540214 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.541795 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:57.541799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:57.541801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:57:57.541816 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:57.541843 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:57:57.541852 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:57.541953 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.542162 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.542244 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:57:57.543581 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.534055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33015 (local address 127.16.40.193:59061)
0812 18:57:57.534886 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.534931 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.536363 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.537026 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.537034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.537135 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:57.537666 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.537681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.538527 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.538530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.539731 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.539742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.540262 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.540266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.540488 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.541323 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:57:57.541325 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:57.541375 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:57:57.541510 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:57.541950 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:57.541962 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:57.541971 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.542127 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.542468 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
I20260812 18:57:57.545048 4773 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33015
I20260812 18:57:57.545329 4773 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:57.545940 4773 heartbeater.cc:511] Master 127.16.40.254:33015 requested a full tablet report, sending...
I20260812 18:57:57.548054 4542 ts_manager.cc:194] Registered new tserver with Master: e6dceddb74364a9999d52912057df18d (127.16.40.193:36627)
I20260812 18:57:57.550103 4542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.193:59061
W20260812 18:57:57.663854 4776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:57.664171 4776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:57.664212 4776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:57.668331 4776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:57.668408 4776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:57.668468 4776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:57:57.673535 4776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:57.674700 4776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:57.676091 4776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:57:57.686460 4776 server_base.cc:1034] running on GCE node
W20260812 18:57:57.686417 4784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.686417 4783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.686417 4786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:57.687064 4776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:57.687829 4776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:57.689023 4776 hybrid_clock.cc:648] HybridClock initialized: now 1786561077689003 us; error 30 us; skew 500 ppm
I20260812 18:57:57.691090 4776 webserver.cc:460] Webserver started at http://127.16.40.194:45317/ using document root <none> and password file <none>
I20260812 18:57:57.691738 4776 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:57.691844 4776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:57.692097 4776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:57.694289 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "d74723fe324a4f49a4760579498ba8b8"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "fb40446c93e2bd425f575f5d40ff2a1b"
server_key_iv: "8967efcf525ebca61f1db3a561dd59f0"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.694890 4776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "d74723fe324a4f49a4760579498ba8b8"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "fb40446c93e2bd425f575f5d40ff2a1b"
server_key_iv: "8967efcf525ebca61f1db3a561dd59f0"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.699082 4776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 18:57:57.702149 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.703806 4776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:57:57.703958 4776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "d74723fe324a4f49a4760579498ba8b8"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "fb40446c93e2bd425f575f5d40ff2a1b"
server_key_iv: "8967efcf525ebca61f1db3a561dd59f0"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.704115 4776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:57.721190 4776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:57.722070 4776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:57.722321 4776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:57.723135 4776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:57.724305 4776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:57.724390 4776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.724507 4776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:57.724586 4776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 18:57:57.734952 4776 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:35773
I20260812 18:57:57.734989 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:35773 every 8 connection(s)
I20260812 18:57:57.735926 4776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:57:57.740464 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4776
I20260812 18:57:57.740645 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:57:57.741025 16547 external_mini_cluster.cc:1428] Setting key d16a6e46b9c89768757d75776ad50031
I20260812 18:57:57.743009 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--builtin_ntp_servers=127.16.40.212:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:57:57.746122 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.738005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:52811 (local address 127.16.40.254:33015)
0812 18:57:57.738187 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.738191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.739353 (+ 1162us) server_negotiation.cc:407] Connection header received
0812 18:57:57.741009 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.741014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.741081 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.741185 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:57.742159 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.742698 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.743963 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.744187 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.744302 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.745641 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:57.745645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:57.745647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:57:57.745660 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:57.745682 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:57:57.745691 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:57.745754 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.745949 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.746005 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 18:57:57.747386 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.738465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33015 (local address 127.16.40.194:52811)
0812 18:57:57.739203 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.739247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.740784 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.741399 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.741406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.741502 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:57.742004 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.742023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.742831 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.742840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.743829 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.743838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.744350 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.744354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.744604 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.745297 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:57:57.745300 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:57.745362 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:57:57.745467 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:57.745780 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:57.745790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:57.745797 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.745948 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.746282 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260812 18:57:57.748840 4933 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33015
I20260812 18:57:57.749137 4933 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:57.749688 4933 heartbeater.cc:511] Master 127.16.40.254:33015 requested a full tablet report, sending...
I20260812 18:57:57.750803 4542 ts_manager.cc:194] Registered new tserver with Master: d74723fe324a4f49a4760579498ba8b8 (127.16.40.194:35773)
I20260812 18:57:57.751377 4542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.194:52811
W20260812 18:57:57.862558 4936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:57.862859 4936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:57.862898 4936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:57.867134 4936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:57.867202 4936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:57.867260 4936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:57:57.872221 4936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:57.873464 4936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:57.874918 4936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:57.885159 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.885246 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:57.885214 4945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:57.885380 4936 server_base.cc:1034] running on GCE node
I20260812 18:57:57.886014 4936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:57.886754 4936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:57.887928 4936 hybrid_clock.cc:648] HybridClock initialized: now 1786561077887885 us; error 54 us; skew 500 ppm
I20260812 18:57:57.889993 4936 webserver.cc:460] Webserver started at http://127.16.40.195:42217/ using document root <none> and password file <none>
I20260812 18:57:57.890625 4936 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:57.890726 4936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:57.890959 4936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:57.893272 4936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "84eec405ada54441aa7c90eb777ab100"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "3d95ba9fc31612c68c1bbccd4c5a83d0"
server_key_iv: "bee7f3fbba6c9dd7c089daea1f2bace7"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.893885 4936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "84eec405ada54441aa7c90eb777ab100"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "3d95ba9fc31612c68c1bbccd4c5a83d0"
server_key_iv: "bee7f3fbba6c9dd7c089daea1f2bace7"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.897837 4936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 18:57:57.900640 4951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.902015 4936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:57:57.902169 4936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "84eec405ada54441aa7c90eb777ab100"
format_stamp: "Formatted at 2026-08-12 18:57:57 on dist-test-slave-rnrw"
server_key: "3d95ba9fc31612c68c1bbccd4c5a83d0"
server_key_iv: "bee7f3fbba6c9dd7c089daea1f2bace7"
server_key_version: "encryptionkey@0"
I20260812 18:57:57.902302 4936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:57.927749 4936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:57.928529 4936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:57.928771 4936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:57.929409 4936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:57:57.930447 4936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:57:57.930543 4936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.930636 4936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:57:57.930689 4936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:57.940078 4936 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:35109
I20260812 18:57:57.940109 5089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:35109 every 8 connection(s)
I20260812 18:57:57.941071 4936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:57:57.941634 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 4936
I20260812 18:57:57.941732 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:57:57.942080 16547 external_mini_cluster.cc:1428] Setting key 17bf90b5e93c38eca63196e76670a9fa
I20260812 18:57:57.950729 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.943150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:35775 (local address 127.16.40.254:33015)
0812 18:57:57.943287 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.943291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.944531 (+ 1240us) server_negotiation.cc:407] Connection header received
0812 18:57:57.945986 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.945989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.946047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.946117 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:57.947054 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.947597 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.948822 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.949038 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.949121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.950225 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:57.950228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:57.950230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:57:57.950243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:57.950263 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:57:57.950271 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:57.950332 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.950558 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.950610 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:57:57.951732 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.943630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33015 (local address 127.16.40.195:35775)
0812 18:57:57.944384 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.944439 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.945777 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.946291 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.946298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.946386 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:57:57.946893 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.946907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.947717 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.947720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.948712 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.948722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.949187 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.949190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.949337 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.949953 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:57:57.949955 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:57:57.950012 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:57:57.950080 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:57:57.950384 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:57:57.950396 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:57:57.950404 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.950539 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.950828 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":142,"threads_started":1}
I20260812 18:57:57.953191 5090 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33015
I20260812 18:57:57.953454 5090 heartbeater.cc:464] Registering TS with master...
I20260812 18:57:57.954085 5090 heartbeater.cc:511] Master 127.16.40.254:33015 requested a full tablet report, sending...
I20260812 18:57:57.955310 4542 ts_manager.cc:194] Registered new tserver with Master: 84eec405ada54441aa7c90eb777ab100 (127.16.40.195:35109)
I20260812 18:57:57.956066 4542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.195:35775
I20260812 18:57:57.956601 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:57:57.962841 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.958820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.16.40.254:33015)
0812 18:57:57.959102 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.959106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.959251 (+ 145us) server_negotiation.cc:407] Connection header received
0812 18:57:57.959418 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.959421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.959469 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.959569 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:57.959853 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.960368 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.961256 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.961496 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.961606 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.962390 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:57.962394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:57.962399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:57:57.962416 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:57.962437 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:57:57.962446 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:57.962534 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.962632 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.962687 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260812 18:57:57.966056 4542 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:57:57.969441 4542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:57:57.986255 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.979648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.16.40.195:35109)
0812 18:57:57.980255 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.980271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.980735 (+ 464us) server_negotiation.cc:407] Connection header received
0812 18:57:57.981082 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.981088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.981307 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.981464 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:57.982018 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.982996 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.984635 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.985504 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.985649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.985870 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.985936 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.986069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":162,"threads_started":1}
I20260812 18:57:57.986569 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.980614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.16.40.193:36627)
0812 18:57:57.981031 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.981047 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.981285 (+ 238us) server_negotiation.cc:407] Connection header received
0812 18:57:57.981454 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.981458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.981618 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.981784 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:57.982118 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.983279 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.985046 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.985946 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.986066 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.986250 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.986307 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.986408 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20260812 18:57:57.986641 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.979429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:35109 (local address 127.0.0.1:44982)
0812 18:57:57.980574 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.980633 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.980870 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.981643 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.981650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.981686 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:57.981899 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.981907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.983135 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.983139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.984401 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.984411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.985669 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.985672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.985755 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.985789 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.985873 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.985998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":74,"spinlock_wait_cycles":5632,"thread_start_us":103,"threads_started":1}
I20260812 18:57:57.987116 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.980174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:36627 (local address 127.0.0.1:51178)
0812 18:57:57.981176 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.981188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.981308 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.981778 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.981781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.981799 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:57.981972 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.981989 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.983419 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.983423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.984904 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.984916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.986084 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.986087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.986167 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.986190 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.986260 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.986331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":233,"thread_start_us":95,"threads_started":1}
I20260812 18:57:57.988299 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.979628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35773 (local address 127.0.0.1:51826)
0812 18:57:57.980794 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:57.980809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:57.980918 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:57.981643 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:57.981650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:57.981687 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:57.981900 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.981907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.983419 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.983423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.984921 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:57.984928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.987850 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.987854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:57.987948 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.987972 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 18:57:57.988054 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 18:57:57.988140 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":212,"spinlock_wait_cycles":8448,"thread_start_us":121,"threads_started":1}
I20260812 18:57:57.988370 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:57.980194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.16.40.194:35773)
0812 18:57:57.980783 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:57.980799 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:57.981067 (+ 268us) server_negotiation.cc:407] Connection header received
0812 18:57:57.981138 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:57.981142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:57.981306 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:57.981464 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:57.982076 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.983279 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.986790 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:57.987710 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:57.987848 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:57.988043 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:57.988100 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 18:57:57.988197 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":128,"threads_started":1}
I20260812 18:57:57.990010 5007 tablet_service.cc:1467] Processing CreateTablet for tablet 80d917a811e147768ce3b5a6e7f47872 (DEFAULT_TABLE table=test-table [id=071d98dc182243a1841e537058f7c94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:57.991163 5007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d917a811e147768ce3b5a6e7f47872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:57.990823 4850 tablet_service.cc:1467] Processing CreateTablet for tablet 80d917a811e147768ce3b5a6e7f47872 (DEFAULT_TABLE table=test-table [id=071d98dc182243a1841e537058f7c94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:57.991864 4850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d917a811e147768ce3b5a6e7f47872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:57.992580 4696 tablet_service.cc:1467] Processing CreateTablet for tablet 80d917a811e147768ce3b5a6e7f47872 (DEFAULT_TABLE table=test-table [id=071d98dc182243a1841e537058f7c94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:57:57.993655 4696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d917a811e147768ce3b5a6e7f47872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:57.998773 5108 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Bootstrap starting.
I20260812 18:57:58.000985 5109 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Bootstrap starting.
I20260812 18:57:58.001062 5110 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Bootstrap starting.
I20260812 18:57:58.002802 5108 tablet_bootstrap.cc:654] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:58.003451 5109 tablet_bootstrap.cc:654] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:58.003679 5110 tablet_bootstrap.cc:654] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:58.004086 5108 log.cc:824] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:58.004333 5109 log.cc:824] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:58.004686 5110 log.cc:824] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Log is configured to *not* fsync() on all Append() calls
I20260812 18:57:58.006706 5109 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: No bootstrap required, opened a new log
I20260812 18:57:58.006723 5108 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: No bootstrap required, opened a new log
I20260812 18:57:58.006933 5110 tablet_bootstrap.cc:492] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: No bootstrap required, opened a new log
I20260812 18:57:58.006973 5109 ts_tablet_manager.cc:1397] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:57:58.007072 5108 ts_tablet_manager.cc:1397] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260812 18:57:58.007154 5110 ts_tablet_manager.cc:1397] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:57:58.011608 5108 raft_consensus.cc:348] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.011950 5108 raft_consensus.cc:374] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:58.012025 5108 raft_consensus.cc:729] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74723fe324a4f49a4760579498ba8b8, State: Initialized, Role: FOLLOWER
I20260812 18:57:58.012130 5109 raft_consensus.cc:348] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.012130 5110 raft_consensus.cc:348] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.012447 5110 raft_consensus.cc:374] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:58.012447 5109 raft_consensus.cc:374] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:58.012519 5110 raft_consensus.cc:729] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dceddb74364a9999d52912057df18d, State: Initialized, Role: FOLLOWER
I20260812 18:57:58.012522 5109 raft_consensus.cc:729] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84eec405ada54441aa7c90eb777ab100, State: Initialized, Role: FOLLOWER
I20260812 18:57:58.012620 5108 consensus_queue.cc:260] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.013019 5110 consensus_queue.cc:260] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.013017 5109 consensus_queue.cc:260] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.013550 4933 heartbeater.cc:503] Master 127.16.40.254:33015 was elected leader, sending a full tablet report...
I20260812 18:57:58.013927 5108 ts_tablet_manager.cc:1428] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 18:57:58.014098 5109 ts_tablet_manager.cc:1428] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:57:58.014098 5110 ts_tablet_manager.cc:1428] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260812 18:57:58.014504 4773 heartbeater.cc:503] Master 127.16.40.254:33015 was elected leader, sending a full tablet report...
I20260812 18:57:58.015558 5090 heartbeater.cc:503] Master 127.16.40.254:33015 was elected leader, sending a full tablet report...
W20260812 18:57:58.032763 4774 tablet.cc:2369] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:57:58.102510 5116 raft_consensus.cc:482] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:57:58.102837 5116 raft_consensus.cc:504] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.104250 5116 leader_election.cc:288] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84eec405ada54441aa7c90eb777ab100 (127.16.40.195:35109), d74723fe324a4f49a4760579498ba8b8 (127.16.40.194:35773)
I20260812 18:57:58.109984 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.104667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:57535 (local address 127.16.40.194:35773)
0812 18:57:58.104818 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:58.104825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:58.105107 (+ 282us) server_negotiation.cc:407] Connection header received
0812 18:57:58.105207 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:58.105210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:58.105267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:58.105342 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:58.105867 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.106735 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.108601 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.109262 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.109397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:58.109452 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:58.109755 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 18:57:58.109836 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 18:57:58.109984 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.104766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:43659 (local address 127.16.40.195:35109)
0812 18:57:58.104899 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:58.104902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:58.105004 (+ 102us) server_negotiation.cc:407] Connection header received
0812 18:57:58.105192 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:58.105196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:58.105254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:58.105342 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:57:58.105886 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.106668 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.108601 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.109262 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.109397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:58.109452 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:58.109755 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 18:57:58.109836 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:57:58.110057 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.104509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:35109 (local address 127.16.40.193:43659)
0812 18:57:58.104906 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:58.104925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:58.105070 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:58.105370 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:58.105373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:58.105405 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:58.105734 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:58.105740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.106946 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.106949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:58.108423 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:58.108434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.109423 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.109428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:58.109554 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:58.109629 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 18:57:58.109722 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 18:57:58.109837 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":176,"spinlock_wait_cycles":19456,"thread_start_us":98,"threads_started":1}
I20260812 18:57:58.110055 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.104582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:35773 (local address 127.16.40.193:57535)
0812 18:57:58.105012 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:58.105024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:57:58.105113 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:57:58.105370 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:57:58.105373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:57:58.105400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:57:58.105731 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:58.105740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.106876 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.106879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:58.108420 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:57:58.108434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.109423 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.109428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:57:58.109573 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:58.109636 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 18:57:58.109722 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 18:57:58.109837 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":163,"spinlock_wait_cycles":896,"thread_start_us":112,"threads_started":1}
I20260812 18:57:58.110870 4873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d917a811e147768ce3b5a6e7f47872" candidate_uuid: "e6dceddb74364a9999d52912057df18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74723fe324a4f49a4760579498ba8b8" is_pre_election: true
I20260812 18:57:58.110860 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d917a811e147768ce3b5a6e7f47872" candidate_uuid: "e6dceddb74364a9999d52912057df18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84eec405ada54441aa7c90eb777ab100" is_pre_election: true
I20260812 18:57:58.111244 4873 raft_consensus.cc:2454] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dceddb74364a9999d52912057df18d in term 0.
I20260812 18:57:58.111244 5031 raft_consensus.cc:2454] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dceddb74364a9999d52912057df18d in term 0.
I20260812 18:57:58.111804 4642 leader_election.cc:302] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d74723fe324a4f49a4760579498ba8b8, e6dceddb74364a9999d52912057df18d; no voters:
I20260812 18:57:58.112265 5116 raft_consensus.cc:2781] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:57:58.112376 5116 raft_consensus.cc:482] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:57:58.112421 5116 raft_consensus.cc:3037] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:58.113600 5116 raft_consensus.cc:504] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.113991 5116 leader_election.cc:288] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [CANDIDATE]: Term 1 election: Requested vote from peers 84eec405ada54441aa7c90eb777ab100 (127.16.40.195:35109), d74723fe324a4f49a4760579498ba8b8 (127.16.40.194:35773)
I20260812 18:57:58.114456 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d917a811e147768ce3b5a6e7f47872" candidate_uuid: "e6dceddb74364a9999d52912057df18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84eec405ada54441aa7c90eb777ab100"
I20260812 18:57:58.114578 4873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d917a811e147768ce3b5a6e7f47872" candidate_uuid: "e6dceddb74364a9999d52912057df18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74723fe324a4f49a4760579498ba8b8"
I20260812 18:57:58.114670 5031 raft_consensus.cc:3037] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:58.114787 4873 raft_consensus.cc:3037] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:58.115813 5031 raft_consensus.cc:2454] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dceddb74364a9999d52912057df18d in term 1.
I20260812 18:57:58.115937 4873 raft_consensus.cc:2454] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dceddb74364a9999d52912057df18d in term 1.
I20260812 18:57:58.116209 4639 leader_election.cc:302] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84eec405ada54441aa7c90eb777ab100, e6dceddb74364a9999d52912057df18d; no voters:
I20260812 18:57:58.116411 5116 raft_consensus.cc:2781] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:58.116703 5116 raft_consensus.cc:686] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 1 LEADER]: Becoming Leader. State: Replica: e6dceddb74364a9999d52912057df18d, State: Running, Role: LEADER
I20260812 18:57:58.117098 5116 consensus_queue.cc:237] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } }
I20260812 18:57:58.120792 4542 catalog_manager.cc:5495] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dceddb74364a9999d52912057df18d (127.16.40.193). New cstate: current_term: 1 leader_uuid: "e6dceddb74364a9999d52912057df18d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dceddb74364a9999d52912057df18d" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 36627 } health_report { overall_health: HEALTHY } } }
I20260812 18:57:58.143347 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.139130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.16.40.193:36627)
0812 18:57:58.139302 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:58.139307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:58.139325 (+ 18us) server_negotiation.cc:407] Connection header received
0812 18:57:58.139528 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:58.139530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:58.139598 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:58.139728 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:57:58.140015 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.140656 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.141633 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:58.141881 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:58.142016 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:58.142382 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:57:58.142866 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:57:58.143035 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:58.143109 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 18:57:58.143175 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 18:57:58.150365 4873 raft_consensus.cc:1261] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 1 FOLLOWER]: Refusing update from remote peer e6dceddb74364a9999d52912057df18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:58.150364 5031 raft_consensus.cc:1261] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 1 FOLLOWER]: Refusing update from remote peer e6dceddb74364a9999d52912057df18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:57:58.151319 5116 consensus_queue.cc:1035] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74723fe324a4f49a4760579498ba8b8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 35773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:58.151571 5119 consensus_queue.cc:1035] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84eec405ada54441aa7c90eb777ab100" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:57:58.170699 4542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:58.170917 4542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:57:58.173674 4542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c67fb1515cc7444bb0b5668f28ca896b: Sending DeleteTablet for 3 replicas of tablet 80d917a811e147768ce3b5a6e7f47872
I20260812 18:57:58.174750 4696 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d917a811e147768ce3b5a6e7f47872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:58 UTC) from {username='slave'} at 127.0.0.1:51178
I20260812 18:57:58.174757 4850 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d917a811e147768ce3b5a6e7f47872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:58 UTC) from {username='slave'} at 127.0.0.1:51826
I20260812 18:57:58.174784 5007 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d917a811e147768ce3b5a6e7f47872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:57:58 UTC) from {username='slave'} at 127.0.0.1:44982
I20260812 18:57:58.175437 5134 tablet_replica.cc:330] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d: stopping tablet replica
I20260812 18:57:58.175833 5134 raft_consensus.cc:2229] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:57:58.176013 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4618
I20260812 18:57:58.176247 5134 raft_consensus.cc:2258] T 80d917a811e147768ce3b5a6e7f47872 P e6dceddb74364a9999d52912057df18d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:58.176743 5133 tablet_replica.cc:330] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: stopping tablet replica
I20260812 18:57:58.176743 5132 tablet_replica.cc:330] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: stopping tablet replica
I20260812 18:57:58.177069 5133 raft_consensus.cc:2229] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:58.177069 5132 raft_consensus.cc:2229] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:57:58.177404 5132 raft_consensus.cc:2258] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:58.177404 5133 raft_consensus.cc:2258] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:57:58.179790 5132 ts_tablet_manager.cc:1905] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:58.179790 5133 ts_tablet_manager.cc:1905] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:57:58.182811 5133 ts_tablet_manager.cc:1918] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:58.182811 5132 ts_tablet_manager.cc:1918] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:57:58.182940 5133 log.cc:1196] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/80d917a811e147768ce3b5a6e7f47872
I20260812 18:57:58.182947 5132 log.cc:1196] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/80d917a811e147768ce3b5a6e7f47872
I20260812 18:57:58.183290 5132 ts_tablet_manager.cc:1939] T 80d917a811e147768ce3b5a6e7f47872 P d74723fe324a4f49a4760579498ba8b8: Deleting consensus metadata
I20260812 18:57:58.183290 5133 ts_tablet_manager.cc:1939] T 80d917a811e147768ce3b5a6e7f47872 P 84eec405ada54441aa7c90eb777ab100: Deleting consensus metadata
I20260812 18:57:58.184868 4520 catalog_manager.cc:4841] TS d74723fe324a4f49a4760579498ba8b8 (127.16.40.194:35773): tablet 80d917a811e147768ce3b5a6e7f47872 (table test-table [id=071d98dc182243a1841e537058f7c94f]) successfully deleted
I20260812 18:57:58.185144 4519 catalog_manager.cc:4841] TS 84eec405ada54441aa7c90eb777ab100 (127.16.40.195:35109): tablet 80d917a811e147768ce3b5a6e7f47872 (table test-table [id=071d98dc182243a1841e537058f7c94f]) successfully deleted
W20260812 18:57:58.190698 4520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:36627 (error 108)
I20260812 18:57:58.190956 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4776
I20260812 18:57:58.194458 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:58.193600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:36627 (local address 127.0.0.1:51188)
0812 18:57:58.194215 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 18:57:58.194319 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:36627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485}
W20260812 18:57:58.194717 4520 catalog_manager.cc:4584] TS e6dceddb74364a9999d52912057df18d (127.16.40.193:36627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80d917a811e147768ce3b5a6e7f47872: Network error: Client connection negotiation failed: client connection to 127.16.40.193:36627: connect: Connection refused (error 111)
I20260812 18:57:58.199874 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4936
I20260812 18:57:58.210299 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 4499
2026-08-12T18:57:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1182 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:57:58 dist-test-slave-rnrw krb5kdc[5138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:57:58 dist-test-slave-rnrw krb5kdc[5138](info): set up 2 sockets
Aug 12 18:57:58 dist-test-slave-rnrw krb5kdc[5138](info): commencing operation
krb5kdc: starting...
W20260812 18:57:59.344389 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:57:59 dist-test-slave-rnrw krb5kdc[5138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:57:59.489056 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:44315
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:59.623988 5152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:59.624326 5152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:59.624372 5152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:59.628731 5152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:57:59.628866 5152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:59.628890 5152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:59.628906 5152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:57:59.628922 5152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:57:59.634616 5152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:59.635793 5152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:59.636847 5152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:59.645949 5160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:59.646049 5161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:59.645963 5163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:59.646252 5152 server_base.cc:1034] running on GCE node
I20260812 18:57:59.646951 5152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:59.648280 5152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:59.649482 5152 hybrid_clock.cc:648] HybridClock initialized: now 1786561079649438 us; error 64 us; skew 500 ppm
Aug 12 18:57:59 dist-test-slave-rnrw krb5kdc[5138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561079, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:59.652439 5152 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:57:59.653437 5152 webserver.cc:460] Webserver started at http://127.16.40.254:43119/ using document root <none> and password file <none>
I20260812 18:57:59.653996 5152 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:59.654055 5152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:59.654229 5152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:59.656333 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "b1335c02f6d34dc7b7883bdc83fe0629"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "d3517d0118fdd7f0451682dfdfb81de0"
server_key_iv: "43d70256d5a1ccd093e4efd98b437e8e"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.657007 5152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "b1335c02f6d34dc7b7883bdc83fe0629"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "d3517d0118fdd7f0451682dfdfb81de0"
server_key_iv: "43d70256d5a1ccd093e4efd98b437e8e"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.661170 5152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:57:59.663782 5172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:59.665359 5152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:59.665478 5152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "b1335c02f6d34dc7b7883bdc83fe0629"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "d3517d0118fdd7f0451682dfdfb81de0"
server_key_iv: "43d70256d5a1ccd093e4efd98b437e8e"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.665591 5152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:59.680853 5152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:57:59.684451 5152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:57:59.684741 5152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:57:59.692651 5152 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:44315
I20260812 18:57:59.692680 5238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:44315 every 8 connection(s)
I20260812 18:57:59.693739 5152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:57:59.696849 5239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:57:59.698403 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5152
I20260812 18:57:59.698534 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:57:59.698872 16547 external_mini_cluster.cc:1428] Setting key f97b572b32d7fdda6f3ca8f5f59237ca
I20260812 18:57:59.703517 5239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: Bootstrap starting.
I20260812 18:57:59.706244 5239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: Neither blocks nor log segments found. Creating new log.
I20260812 18:57:59.707374 5239 log.cc:824] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:57:59 dist-test-slave-rnrw krb5kdc[5138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:57:59.710129 5239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: No bootstrap required, opened a new log
I20260812 18:57:59.715397 5239 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } }
I20260812 18:57:59.715747 5239 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:57:59.715812 5239 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1335c02f6d34dc7b7883bdc83fe0629, State: Initialized, Role: FOLLOWER
I20260812 18:57:59.716279 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:57:59.700535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.16.40.254:44315)
0812 18:57:59.701071 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0812 18:57:59.701089 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:57:59.701131 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:57:59.701835 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:57:59.701854 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:57:59.702238 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:57:59.702615 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:57:59.703543 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:59.704686 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:59.705617 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:57:59.706318 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:57:59.706517 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:57:59.710089 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:57:59.710096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:57:59.710116 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:57:59.710156 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:57:59.712306 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:59.712966 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:59.712976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:59.712986 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:59.713166 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:57:59.713530 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:57:59.713534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:57:59.713536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:57:59.714157 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:57:59.714380 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 18:57:59.714751 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 18:57:59.715133 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260812 18:57:59.716343 5239 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } }
I20260812 18:57:59.716501 5239 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:57:59.716596 5239 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:57:59.716710 5239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:57:59.717790 5239 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } }
I20260812 18:57:59.718209 5239 leader_election.cc:302] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1335c02f6d34dc7b7883bdc83fe0629; no voters:
I20260812 18:57:59.718604 5239 leader_election.cc:288] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:57:59.718768 5245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:57:59.719029 5245 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [term 1 LEADER]: Becoming Leader. State: Replica: b1335c02f6d34dc7b7883bdc83fe0629, State: Running, Role: LEADER
I20260812 18:57:59.719391 5245 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } }
I20260812 18:57:59.720014 5239 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:57:59.721413 5246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1335c02f6d34dc7b7883bdc83fe0629. Latest consensus state: current_term: 1 leader_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } } }
I20260812 18:57:59.721588 5246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:59.721864 5247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1335c02f6d34dc7b7883bdc83fe0629" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 44315 } } }
I20260812 18:57:59.722064 5254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:57:59.722079 5247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629 [sys.catalog]: This master's current role is: LEADER
I20260812 18:57:59.726496 5254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:57:59.733150 5254 catalog_manager.cc:1347] Generated new cluster ID: 604f97529fc44736acded348390427ec
I20260812 18:57:59.733299 5254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:57:59.747576 5254 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:57:59.749836 5254 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:57:59.757112 5254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: Generated new TSK 0
I20260812 18:57:59.757936 5254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:57:59.794291 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:57:59.920203 5268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:57:59.920642 5268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:57:59.920765 5268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:57:59.925222 5268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:57:59.925347 5268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:57:59.925457 5268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:57:59.931468 5268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:57:59.932734 5268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:57:59.933849 5268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:57:59.945087 5278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:57:59.945011 5273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:59.945269 5268 server_base.cc:1034] running on GCE node
W20260812 18:57:59.945017 5274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:57:59.945858 5268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:57:59.946710 5268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:57:59.947898 5268 hybrid_clock.cc:648] HybridClock initialized: now 1786561079947877 us; error 35 us; skew 500 ppm
Aug 12 18:57:59 dist-test-slave-rnrw krb5kdc[5138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561079, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:57:59.951333 5268 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:57:59.952354 5268 webserver.cc:460] Webserver started at http://127.16.40.193:38559/ using document root <none> and password file <none>
I20260812 18:57:59.953045 5268 fs_manager.cc:359] Metadata directory not provided
I20260812 18:57:59.953112 5268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:57:59.953297 5268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:57:59.955471 5268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "ce702ac7bc1d4fbaa3d01e368417e859"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "edc5ef179278283e00afa939db6e2fdd"
server_key_iv: "0a227c2b7812fc4b2a3d9895c16d29bd"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.956034 5268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "ce702ac7bc1d4fbaa3d01e368417e859"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "edc5ef179278283e00afa939db6e2fdd"
server_key_iv: "0a227c2b7812fc4b2a3d9895c16d29bd"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.960476 5268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260812 18:57:59.963389 5285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:57:59.964861 5268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:57:59.965042 5268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "ce702ac7bc1d4fbaa3d01e368417e859"
format_stamp: "Formatted at 2026-08-12 18:57:59 on dist-test-slave-rnrw"
server_key: "edc5ef179278283e00afa939db6e2fdd"
server_key_iv: "0a227c2b7812fc4b2a3d9895c16d29bd"
server_key_version: "encryptionkey@0"
I20260812 18:57:59.965178 5268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:57:59.996354 5268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:00.000583 5268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:00.000885 5268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:00.001617 5268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:00.002632 5268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:00.002703 5268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.002779 5268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:00.002810 5268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.012599 5268 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43795
I20260812 18:58:00.012633 5426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43795 every 8 connection(s)
I20260812 18:58:00.013767 5268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:58:00.023777 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5268
I20260812 18:58:00.023952 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:58:00.024358 16547 external_mini_cluster.cc:1428] Setting key c7efc53db85202142a858313f14405f7
Aug 12 18:58:00 dist-test-slave-rnrw krb5kdc[5138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561079, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:00.029670 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.015732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:39767 (local address 127.16.40.254:44315)
0812 18:58:00.015892 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.015896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.016798 (+ 902us) server_negotiation.cc:407] Connection header received
0812 18:58:00.018358 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.018362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.018433 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.018551 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:00.020084 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.020764 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.022002 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.022297 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.022424 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.025562 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:00.025568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:00.025575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:00.025604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:00.027219 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.027833 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.027837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.027839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.027932 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.028336 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.028340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.028342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.028612 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:00.028735 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.029320 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.029480 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:58:00.030584 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.016067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44315 (local address 127.16.40.193:39767)
0812 18:58:00.016607 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.016673 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.018070 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.018759 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.018767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.019170 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:00.019920 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.019938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.020903 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.020908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.021861 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.021871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.022476 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.022480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.022681 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.023307 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:00.023326 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:00.025345 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:00.027364 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.027372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.027393 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.027714 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.028036 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.028039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.028041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.028224 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.028736 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:00.028746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:00.029051 (+ 305us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.029275 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.029657 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260812 18:58:00.032290 5427 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44315
I20260812 18:58:00.032681 5427 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:00.033325 5427 heartbeater.cc:511] Master 127.16.40.254:44315 requested a full tablet report, sending...
I20260812 18:58:00.035498 5186 ts_manager.cc:194] Registered new tserver with Master: ce702ac7bc1d4fbaa3d01e368417e859 (127.16.40.193:43795)
I20260812 18:58:00.037484 5186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:39767
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:58:00.060514 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:00.183126 5434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:00.183439 5434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:00.183486 5434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:00.187835 5434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:00.187922 5434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:00.187986 5434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:58:00.193636 5434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:00.194825 5434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:00.195915 5434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:00.207170 5439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:00.207255 5434 server_base.cc:1034] running on GCE node
W20260812 18:58:00.207185 5440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:00.207185 5443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:00.208180 5434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:00.209080 5434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:00.210282 5434 hybrid_clock.cc:648] HybridClock initialized: now 1786561080210264 us; error 33 us; skew 500 ppm
Aug 12 18:58:00 dist-test-slave-rnrw krb5kdc[5138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:00.213666 5434 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:58:00.214802 5434 webserver.cc:460] Webserver started at http://127.16.40.194:37451/ using document root <none> and password file <none>
I20260812 18:58:00.215480 5434 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:00.215553 5434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:00.215811 5434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:00.218214 5434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "b75c05daf60b4e75ae131acbc920a889"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "aed12110de878705af8fae12fb1036fd"
server_key_iv: "e51cbe4f72dc9d52c381f67b841db2e8"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.218927 5434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "b75c05daf60b4e75ae131acbc920a889"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "aed12110de878705af8fae12fb1036fd"
server_key_iv: "e51cbe4f72dc9d52c381f67b841db2e8"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.223527 5434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:58:00.226616 5451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.228154 5434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:58:00.228276 5434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "b75c05daf60b4e75ae131acbc920a889"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "aed12110de878705af8fae12fb1036fd"
server_key_iv: "e51cbe4f72dc9d52c381f67b841db2e8"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.228415 5434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:00.251348 5434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:00.255415 5434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:00.255686 5434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:00.256590 5434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:00.257706 5434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:00.257788 5434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.257874 5434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:00.257933 5434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.268743 5434 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:33005
I20260812 18:58:00.268813 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:33005 every 8 connection(s)
I20260812 18:58:00.270018 5434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:58:00.279572 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5434
I20260812 18:58:00.279736 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:58:00.280124 16547 external_mini_cluster.cc:1428] Setting key 84fb0b3af4adad2f85a58438d13a1cd7
Aug 12 18:58:00 dist-test-slave-rnrw krb5kdc[5138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:00.287204 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.272243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:34239 (local address 127.16.40.254:44315)
0812 18:58:00.272419 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.272425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.273434 (+ 1009us) server_negotiation.cc:407] Connection header received
0812 18:58:00.275198 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.275203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.275280 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.275390 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:00.276850 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.277490 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.278760 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.279054 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.279165 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.282541 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:00.282545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:00.282547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:00.282576 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:00.284296 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.285160 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.285163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.285164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.285256 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.285777 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.285780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.285782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.285997 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:00.286093 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.286936 (+ 843us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.287069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 18:58:00.288908 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.272714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44315 (local address 127.16.40.194:34239)
0812 18:58:00.273251 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.273301 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.274886 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.275615 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.275623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.276037 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:00.276659 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.276683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.277650 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.277654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.278607 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.278618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.279238 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.279240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.279408 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.280161 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:00.280180 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:00.282309 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:00.284443 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.284454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.284479 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.285011 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.285367 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.285370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.285373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.285638 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.286132 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:00.286144 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:00.286600 (+ 456us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.286858 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.287325 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260812 18:58:00.290498 5590 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44315
I20260812 18:58:00.290860 5590 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:00.291456 5590 heartbeater.cc:511] Master 127.16.40.254:44315 requested a full tablet report, sending...
I20260812 18:58:00.292847 5186 ts_manager.cc:194] Registered new tserver with Master: b75c05daf60b4e75ae131acbc920a889 (127.16.40.194:33005)
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:58:00.293591 5186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:34239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:58:00.316015 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:00.443645 5596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:00.443943 5596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:00.444015 5596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:00.448863 5596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:00.448957 5596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:00.449059 5596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:58:00.455232 5596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:44315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:00.456501 5596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:00.457841 5596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:00.468631 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:00.468683 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:00.468771 5596 server_base.cc:1034] running on GCE node
W20260812 18:58:00.468947 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:00.469386 5596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:00.470212 5596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:00.471421 5596 hybrid_clock.cc:648] HybridClock initialized: now 1786561080471391 us; error 44 us; skew 500 ppm
Aug 12 18:58:00 dist-test-slave-rnrw krb5kdc[5138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:00.474773 5596 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:58:00.475833 5596 webserver.cc:460] Webserver started at http://127.16.40.195:33395/ using document root <none> and password file <none>
I20260812 18:58:00.476475 5596 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:00.476599 5596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:00.476857 5596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:00.479076 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "3d331f51f612443ba9004ff3af86cc31"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "b3b36a9a820e9433d48e6ae10ebe9dee"
server_key_iv: "069bf52e688b0654800900c7a05a5767"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.479698 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "3d331f51f612443ba9004ff3af86cc31"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "b3b36a9a820e9433d48e6ae10ebe9dee"
server_key_iv: "069bf52e688b0654800900c7a05a5767"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.483961 5596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:58:00.486954 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.488372 5596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 18:58:00.488643 5596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "3d331f51f612443ba9004ff3af86cc31"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "b3b36a9a820e9433d48e6ae10ebe9dee"
server_key_iv: "069bf52e688b0654800900c7a05a5767"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.488786 5596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:00.506280 5596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:00.510000 5596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:00.510245 5596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:00.510939 5596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:00.512061 5596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:00.512164 5596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.512247 5596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:00.512306 5596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.523212 5596 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:42805
I20260812 18:58:00.523234 5758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:42805 every 8 connection(s)
I20260812 18:58:00.524286 5596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:58:00.524967 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5596
I20260812 18:58:00.525125 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:58:00.525467 16547 external_mini_cluster.cc:1428] Setting key 999940b0a824be19fea440cb2494b7c4
Aug 12 18:58:00 dist-test-slave-rnrw krb5kdc[5138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:00.540916 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.526668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:48009 (local address 127.16.40.254:44315)
0812 18:58:00.526833 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.526836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.527926 (+ 1090us) server_negotiation.cc:407] Connection header received
0812 18:58:00.529504 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.529507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.529566 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.529681 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:00.530943 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.531604 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.532981 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.533221 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.533378 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.536292 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:00.536296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:00.536297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:00.536327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:00.538013 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.538769 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.538774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.538776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.538898 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.539418 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.539421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.539422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.539692 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:00.539831 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.540661 (+ 830us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.540790 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:58:00.542012 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.527179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:44315 (local address 127.16.40.195:48009)
0812 18:58:00.527760 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.527809 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.529284 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.529868 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.529875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.530266 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:00.530786 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.530800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.531756 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.531759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.532847 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.532858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.533472 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.533477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.533654 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.534253 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:00.534272 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:00.536070 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:00.538182 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.538190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.538211 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.538619 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.539070 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:00.539074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:00.539076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:00.539294 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:00.539863 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:00.539871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:00.540250 (+ 379us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.540609 (+ 359us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.540947 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":179,"threads_started":1}
I20260812 18:58:00.543272 5759 heartbeater.cc:347] Connected to a master server at 127.16.40.254:44315
I20260812 18:58:00.543546 5759 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:00.544118 5759 heartbeater.cc:511] Master 127.16.40.254:44315 requested a full tablet report, sending...
I20260812 18:58:00.545331 5186 ts_manager.cc:194] Registered new tserver with Master: 3d331f51f612443ba9004ff3af86cc31 (127.16.40.195:42805)
I20260812 18:58:00.546056 5186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:48009
I20260812 18:58:00.552500 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:58:00.563488 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.554645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.16.40.254:44315)
0812 18:58:00.554824 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.554829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.554994 (+ 165us) server_negotiation.cc:407] Connection header received
0812 18:58:00.555131 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.555134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.555180 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.555253 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:00.556024 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.556714 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.557628 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.557847 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.557936 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.559756 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:00.559759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:00.559761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:00.559798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:00.561416 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.562048 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.562051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.562053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.562143 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:00.562456 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:00.562458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:00.562460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:00.562746 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:00.562872 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.563191 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.563285 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 18:58:00.567191 5186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:58:00.570927 5186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:58:00.588083 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.581873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.16.40.195:42805)
0812 18:58:00.582396 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.582409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.582432 (+ 23us) server_negotiation.cc:407] Connection header received
0812 18:58:00.582507 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.582512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.582699 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.582855 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:00.583304 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584385 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.586493 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587235 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.587414 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.587591 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.587785 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.587903 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":178,"threads_started":1}
I20260812 18:58:00.588102 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.581120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:42805 (local address 127.0.0.1:42930)
0812 18:58:00.581556 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.581571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.581698 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.583036 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.583040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.583060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:00.583161 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.583167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584613 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.584616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.586377 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.586384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587416 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.587420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.587556 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.587599 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.587748 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.587897 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":263,"spinlock_wait_cycles":5376,"thread_start_us":137,"threads_started":1}
I20260812 18:58:00.588272 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.581108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.16.40.194:33005)
0812 18:58:00.581993 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.582008 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.582050 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:58:00.582136 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.582144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.582387 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.582554 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:00.583059 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584262 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.586261 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587274 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.587386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.587584 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.587990 (+ 406us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.588103 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":167,"threads_started":1}
I20260812 18:58:00.588105 5773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.581761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43795 (local address 127.0.0.1:57810)
0812 18:58:00.582086 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.582101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.582214 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.583422 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.583425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.583444 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:00.583540 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.583547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584512 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.584518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.586009 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.586023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587429 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.587432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.587585 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.587649 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.587705 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.587892 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":199,"thread_start_us":86,"threads_started":1}
I20260812 18:58:00.588673 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.580962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33005 (local address 127.0.0.1:50958)
0812 18:58:00.581472 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.581544 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.581698 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.582712 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.582720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.582752 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:00.582928 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.582940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584416 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.584421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.586154 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.586165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587654 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.587657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.587746 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.587766 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.587831 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.587892 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":130,"spinlock_wait_cycles":2304,"thread_start_us":117,"threads_started":1}
I20260812 18:58:00.589629 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.582398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.16.40.193:43795)
0812 18:58:00.583000 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.583008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.583029 (+ 21us) server_negotiation.cc:407] Connection header received
0812 18:58:00.583098 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.583103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.583298 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.583446 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:00.583643 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.584374 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.586163 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.587244 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.588242 (+ 998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.588829 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.589198 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.589320 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":74,"threads_started":1}
I20260812 18:58:00.592041 5507 tablet_service.cc:1467] Processing CreateTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 (DEFAULT_TABLE table=test-table [id=ac391ee4eccf4cd3b1a310c6300a142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:00.592077 5670 tablet_service.cc:1467] Processing CreateTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 (DEFAULT_TABLE table=test-table [id=ac391ee4eccf4cd3b1a310c6300a142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:00.592059 5343 tablet_service.cc:1467] Processing CreateTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 (DEFAULT_TABLE table=test-table [id=ac391ee4eccf4cd3b1a310c6300a142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:00.593286 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70179b4c849849d5a776e0cd1a80efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:00.593286 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70179b4c849849d5a776e0cd1a80efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:00.593287 5507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70179b4c849849d5a776e0cd1a80efb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:00.600898 5777 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: Bootstrap starting.
I20260812 18:58:00.602180 5778 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: Bootstrap starting.
I20260812 18:58:00.602586 5779 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Bootstrap starting.
I20260812 18:58:00.603788 5777 tablet_bootstrap.cc:654] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:00.604343 5778 tablet_bootstrap.cc:654] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:00.604949 5779 tablet_bootstrap.cc:654] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:00.605337 5778 log.cc:824] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:00.605369 5777 log.cc:824] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:00.606377 5779 log.cc:824] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:00.607848 5777 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: No bootstrap required, opened a new log
I20260812 18:58:00.607848 5778 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: No bootstrap required, opened a new log
I20260812 18:58:00.608098 5777 ts_tablet_manager.cc:1397] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 18:58:00.608098 5778 ts_tablet_manager.cc:1397] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:58:00.608462 5779 tablet_bootstrap.cc:492] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: No bootstrap required, opened a new log
I20260812 18:58:00.608724 5779 ts_tablet_manager.cc:1397] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:58:00.611757 5779 raft_consensus.cc:348] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.612104 5779 raft_consensus.cc:374] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:00.612170 5779 raft_consensus.cc:729] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d331f51f612443ba9004ff3af86cc31, State: Initialized, Role: FOLLOWER
I20260812 18:58:00.612394 5777 raft_consensus.cc:348] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.612725 5777 raft_consensus.cc:374] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:00.612782 5777 raft_consensus.cc:729] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75c05daf60b4e75ae131acbc920a889, State: Initialized, Role: FOLLOWER
I20260812 18:58:00.612792 5779 consensus_queue.cc:260] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.612843 5778 raft_consensus.cc:348] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.613109 5778 raft_consensus.cc:374] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:00.613159 5778 raft_consensus.cc:729] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce702ac7bc1d4fbaa3d01e368417e859, State: Initialized, Role: FOLLOWER
I20260812 18:58:00.613281 5777 consensus_queue.cc:260] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.613579 5778 consensus_queue.cc:260] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.614135 5779 ts_tablet_manager.cc:1428] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 18:58:00.614336 5777 ts_tablet_manager.cc:1428] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260812 18:58:00.614418 5759 heartbeater.cc:503] Master 127.16.40.254:44315 was elected leader, sending a full tablet report...
I20260812 18:58:00.614459 5778 ts_tablet_manager.cc:1428] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260812 18:58:00.614761 5427 heartbeater.cc:503] Master 127.16.40.254:44315 was elected leader, sending a full tablet report...
I20260812 18:58:00.615164 5590 heartbeater.cc:503] Master 127.16.40.254:44315 was elected leader, sending a full tablet report...
I20260812 18:58:00.616426 5785 raft_consensus.cc:482] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:58:00.616608 5785 raft_consensus.cc:504] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.618691 5785 leader_election.cc:288] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d331f51f612443ba9004ff3af86cc31 (127.16.40.195:42805), ce702ac7bc1d4fbaa3d01e368417e859 (127.16.40.193:43795)
I20260812 18:58:00.622785 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.618815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:40165 (local address 127.16.40.195:42805)
0812 18:58:00.618921 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.618926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.618941 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:58:00.618991 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.618994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.619038 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.619131 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:00.619433 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.620079 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.621479 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.622138 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.622299 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.622361 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.622604 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.622684 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 18:58:00.622790 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.618064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:42805 (local address 127.16.40.194:40165)
0812 18:58:00.618328 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.618346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.618433 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.619136 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.619140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.619159 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:00.619294 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.619305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.620262 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.620266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.621305 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.621320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.622273 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.622277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.622414 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.622444 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.622552 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.622639 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260812 18:58:00.623483 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.618993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:46279 (local address 127.16.40.193:43795)
0812 18:58:00.619145 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.619150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.619162 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:58:00.619207 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.619209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.619252 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.619337 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:00.619742 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.620522 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.621919 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.622472 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.622609 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.622655 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.623316 (+ 661us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.623391 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 18:58:00.623543 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70179b4c849849d5a776e0cd1a80efb8" candidate_uuid: "b75c05daf60b4e75ae131acbc920a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d331f51f612443ba9004ff3af86cc31" is_pre_election: true
I20260812 18:58:00.623492 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.618444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43795 (local address 127.16.40.194:46279)
0812 18:58:00.618857 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.618875 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:00.618977 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:00.619377 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:00.619380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:00.619401 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:00.619600 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.619609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.620729 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.620732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.621804 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:00.621815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.622718 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.622721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:00.622822 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.623197 (+ 375us) client_negotiation.cc:769] Sending connection context
0812 18:58:00.623286 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 18:58:00.623359 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260812 18:58:00.623916 5695 raft_consensus.cc:2454] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75c05daf60b4e75ae131acbc920a889 in term 0.
I20260812 18:58:00.624271 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70179b4c849849d5a776e0cd1a80efb8" candidate_uuid: "b75c05daf60b4e75ae131acbc920a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" is_pre_election: true
I20260812 18:58:00.624536 5369 raft_consensus.cc:2454] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75c05daf60b4e75ae131acbc920a889 in term 0.
I20260812 18:58:00.624528 5454 leader_election.cc:302] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d331f51f612443ba9004ff3af86cc31, b75c05daf60b4e75ae131acbc920a889; no voters:
I20260812 18:58:00.624878 5785 raft_consensus.cc:2781] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:58:00.624990 5785 raft_consensus.cc:482] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:58:00.625057 5785 raft_consensus.cc:3037] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:00.626179 5785 raft_consensus.cc:504] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.626581 5785 leader_election.cc:288] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [CANDIDATE]: Term 1 election: Requested vote from peers 3d331f51f612443ba9004ff3af86cc31 (127.16.40.195:42805), ce702ac7bc1d4fbaa3d01e368417e859 (127.16.40.193:43795)
I20260812 18:58:00.627053 5695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70179b4c849849d5a776e0cd1a80efb8" candidate_uuid: "b75c05daf60b4e75ae131acbc920a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d331f51f612443ba9004ff3af86cc31"
I20260812 18:58:00.627130 5369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70179b4c849849d5a776e0cd1a80efb8" candidate_uuid: "b75c05daf60b4e75ae131acbc920a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce702ac7bc1d4fbaa3d01e368417e859"
I20260812 18:58:00.627204 5695 raft_consensus.cc:3037] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:00.627295 5369 raft_consensus.cc:3037] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:00.628404 5695 raft_consensus.cc:2454] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75c05daf60b4e75ae131acbc920a889 in term 1.
I20260812 18:58:00.628504 5369 raft_consensus.cc:2454] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75c05daf60b4e75ae131acbc920a889 in term 1.
I20260812 18:58:00.628809 5454 leader_election.cc:302] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d331f51f612443ba9004ff3af86cc31, b75c05daf60b4e75ae131acbc920a889; no voters:
I20260812 18:58:00.629011 5785 raft_consensus.cc:2781] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:00.629308 5785 raft_consensus.cc:686] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 1 LEADER]: Becoming Leader. State: Replica: b75c05daf60b4e75ae131acbc920a889, State: Running, Role: LEADER
I20260812 18:58:00.629709 5785 consensus_queue.cc:237] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } }
I20260812 18:58:00.633291 5188 catalog_manager.cc:5495] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 reported cstate change: term changed from 0 to 1, leader changed from <none> to b75c05daf60b4e75ae131acbc920a889 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "b75c05daf60b4e75ae131acbc920a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75c05daf60b4e75ae131acbc920a889" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 33005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 } health_report { overall_health: UNKNOWN } } }
I20260812 18:58:00.658042 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.653848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.16.40.194:33005)
0812 18:58:00.654019 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.654023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.654059 (+ 36us) server_negotiation.cc:407] Connection header received
0812 18:58:00.654194 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.654200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.654284 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.654391 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:00.654745 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.655279 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.656297 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.656519 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.656691 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.656994 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:00.657462 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:00.657587 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.657769 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.657841 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 18:58:00.665256 5695 raft_consensus.cc:1261] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 1 FOLLOWER]: Refusing update from remote peer b75c05daf60b4e75ae131acbc920a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:00.665257 5369 raft_consensus.cc:1261] T 70179b4c849849d5a776e0cd1a80efb8 P ce702ac7bc1d4fbaa3d01e368417e859 [term 1 FOLLOWER]: Refusing update from remote peer b75c05daf60b4e75ae131acbc920a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:00.666057 5785 consensus_queue.cc:1035] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d331f51f612443ba9004ff3af86cc31" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:00.666339 5789 consensus_queue.cc:1035] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce702ac7bc1d4fbaa3d01e368417e859" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:00.676461 5794 mvcc.cc:205] Tried to move back new op lower bound from 7317754186396528640 to 7317754186262167552. Current Snapshot: MvccSnapshot[applied={T|T < 7317754186396528640}]
I20260812 18:58:00.689757 5188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:00.689993 5188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:00.693226 5188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1335c02f6d34dc7b7883bdc83fe0629: Sending DeleteTablet for 3 replicas of tablet 70179b4c849849d5a776e0cd1a80efb8
I20260812 18:58:00.694237 5507 tablet_service.cc:1514] Processing DeleteTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:00 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:50958
I20260812 18:58:00.694856 5803 tablet_replica.cc:330] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889: stopping tablet replica
I20260812 18:58:00.694877 5343 tablet_service.cc:1514] Processing DeleteTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:00 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:57810
I20260812 18:58:00.695366 5803 raft_consensus.cc:2229] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:58:00.695850 5803 raft_consensus.cc:2258] T 70179b4c849849d5a776e0cd1a80efb8 P b75c05daf60b4e75ae131acbc920a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:00.696051 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5268
I20260812 18:58:00.697798 5670 tablet_service.cc:1514] Processing DeleteTablet for tablet 70179b4c849849d5a776e0cd1a80efb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:00 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:42930
I20260812 18:58:00.706213 5806 tablet_replica.cc:330] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: stopping tablet replica
W20260812 18:58:00.706368 5176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:43795 (error 108)
I20260812 18:58:00.706678 5806 raft_consensus.cc:2229] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:00.707105 5806 raft_consensus.cc:2258] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:00.708860 5806 ts_tablet_manager.cc:1905] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:00.709187 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5434
I20260812 18:58:00.712962 5806 ts_tablet_manager.cc:1918] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:00.713138 5806 log.cc:1196] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/70179b4c849849d5a776e0cd1a80efb8
I20260812 18:58:00.713613 5806 ts_tablet_manager.cc:1939] T 70179b4c849849d5a776e0cd1a80efb8 P 3d331f51f612443ba9004ff3af86cc31: Deleting consensus metadata
I20260812 18:58:00.715602 5175 catalog_manager.cc:4841] TS 3d331f51f612443ba9004ff3af86cc31 (127.16.40.195:42805): tablet 70179b4c849849d5a776e0cd1a80efb8 (table test-table [id=ac391ee4eccf4cd3b1a310c6300a142e]) successfully deleted
I20260812 18:58:00.717870 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.717506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43795 (local address 127.0.0.1:57812)
0812 18:58:00.717686 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.717774 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 18:58:00.718137 5176 catalog_manager.cc:4584] TS ce702ac7bc1d4fbaa3d01e368417e859 (127.16.40.193:43795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70179b4c849849d5a776e0cd1a80efb8: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43795: connect: Connection refused (error 111)
I20260812 18:58:00.724156 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5596
I20260812 18:58:00.727579 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.724310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:33005 (local address 127.0.0.1:50972)
0812 18:58:00.727320 (+ 3010us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:00.727431 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.194:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2858}
I20260812 18:58:00.733799 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5152
2026-08-12T18:58:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2526 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T18:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:00Z Disabled control of system clock
I20260812 18:58:00.773250 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35133
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:35133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:58:00.895712 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:00.896027 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:00.896065 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:00.900337 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:00.900434 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:00.900449 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:00.900467 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:00.900489 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:00.905733 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:35133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:00.907069 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:00.908654 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:00.917982 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:00.918011 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:00.917944 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:00.918401 5812 server_base.cc:1034] running on GCE node
I20260812 18:58:00.919093 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:00.920360 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:00.921547 5812 hybrid_clock.cc:648] HybridClock initialized: now 1786561080921530 us; error 31 us; skew 500 ppm
I20260812 18:58:00.923909 5812 webserver.cc:460] Webserver started at http://127.16.40.254:37913/ using document root <none> and password file <none>
I20260812 18:58:00.924654 5812 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:00.924731 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:00.924985 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:00.927282 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "7613243322f9408c899a57844b807535"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "e2e8131ddd3c25d42f5bb55c130d3f47"
server_key_iv: "e0860a2c62598e9bb22d108319a7ba49"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.927966 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "7613243322f9408c899a57844b807535"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "e2e8131ddd3c25d42f5bb55c130d3f47"
server_key_iv: "e0860a2c62598e9bb22d108319a7ba49"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.932303 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:00.935279 5831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:00.936847 5812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:00.937040 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "7613243322f9408c899a57844b807535"
format_stamp: "Formatted at 2026-08-12 18:58:00 on dist-test-slave-rnrw"
server_key: "e2e8131ddd3c25d42f5bb55c130d3f47"
server_key_iv: "e0860a2c62598e9bb22d108319a7ba49"
server_key_version: "encryptionkey@0"
I20260812 18:58:00.937182 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:00.957604 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:00.958484 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:00.958737 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:00.967244 5812 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:35133
I20260812 18:58:00.967267 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:35133 every 8 connection(s)
I20260812 18:58:00.968344 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:00.972030 5896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:00.972291 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5812
I20260812 18:58:00.972456 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:00.972855 16547 external_mini_cluster.cc:1428] Setting key c8c23937f7160ffe05719f763927156d
I20260812 18:58:00.979723 5896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: Bootstrap starting.
I20260812 18:58:00.982383 5896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:00.982844 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:00.974500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.16.40.254:35133)
0812 18:58:00.975258 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:00.975275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:00.975312 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:58:00.976027 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:00.976047 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:00.976481 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:00.976915 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:00.977384 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.978382 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.979313 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:00.979820 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:00.980006 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:00.980972 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:00.980977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:00.980991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:00.981399 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:00.981460 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:00.981481 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:00.981593 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:00.981760 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 18:58:00.982036 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":161,"threads_started":1}
I20260812 18:58:00.983304 5896 log.cc:824] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:00.985636 5896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: No bootstrap required, opened a new log
I20260812 18:58:00.989666 5896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } }
I20260812 18:58:00.989951 5896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:00.989997 5896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7613243322f9408c899a57844b807535, State: Initialized, Role: FOLLOWER
I20260812 18:58:00.990449 5896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } }
I20260812 18:58:00.990594 5896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:00.990667 5896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:00.990797 5896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:00.991803 5896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } }
I20260812 18:58:00.992180 5896 leader_election.cc:302] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7613243322f9408c899a57844b807535; no voters:
I20260812 18:58:00.992619 5896 leader_election.cc:288] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:00.992777 5901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:00.993036 5901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [term 1 LEADER]: Becoming Leader. State: Replica: 7613243322f9408c899a57844b807535, State: Running, Role: LEADER
I20260812 18:58:00.993366 5901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } }
I20260812 18:58:00.994057 5896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:00.995364 5902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7613243322f9408c899a57844b807535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } } }
I20260812 18:58:00.995515 5902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:00.995764 5903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7613243322f9408c899a57844b807535. Latest consensus state: current_term: 1 leader_uuid: "7613243322f9408c899a57844b807535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7613243322f9408c899a57844b807535" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 35133 } } }
I20260812 18:58:00.995870 5903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:00.996069 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:00.999194 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:01.006084 5913 catalog_manager.cc:1347] Generated new cluster ID: 80831fb0d14e4eb085ea3774eef8a10e
I20260812 18:58:01.006202 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:01.018036 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:01.019428 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:01.029901 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: Generated new TSK 0
I20260812 18:58:01.030921 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:01.034578 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--builtin_ntp_servers=127.16.40.212:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:58:01.165937 5925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:01.166268 5925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:01.166311 5925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:01.170670 5925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:01.170759 5925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:01.170821 5925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:58:01.175947 5925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:01.177201 5925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:01.178733 5925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:01.189776 5932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:01.189852 5930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:01.189788 5934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:01.190305 5925 server_base.cc:1034] running on GCE node
I20260812 18:58:01.190843 5925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:01.191603 5925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:01.192847 5925 hybrid_clock.cc:648] HybridClock initialized: now 1786561081192829 us; error 39 us; skew 500 ppm
I20260812 18:58:01.195508 5925 webserver.cc:460] Webserver started at http://127.16.40.193:43983/ using document root <none> and password file <none>
I20260812 18:58:01.196321 5925 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:01.196453 5925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:01.196754 5925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:01.199290 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "7735d92f8eff415cbb2185bc58e02874"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "6adb50be2853964e7ad8845f0255b01e"
server_key_iv: "771b70cbeaf30cf0be7b2f77a8ed1af3"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.199940 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "7735d92f8eff415cbb2185bc58e02874"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "6adb50be2853964e7ad8845f0255b01e"
server_key_iv: "771b70cbeaf30cf0be7b2f77a8ed1af3"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.204528 5925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:01.207688 5941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.209374 5925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:01.209507 5925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "7735d92f8eff415cbb2185bc58e02874"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "6adb50be2853964e7ad8845f0255b01e"
server_key_iv: "771b70cbeaf30cf0be7b2f77a8ed1af3"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.209637 5925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:01.233774 5925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:01.234603 5925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:01.234792 5925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:01.235431 5925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:01.236533 5925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:01.236613 5925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.236658 5925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:01.236675 5925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.247462 5925 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:43585
I20260812 18:58:01.247509 6078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:43585 every 8 connection(s)
I20260812 18:58:01.248622 5925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:58:01.256922 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 5925
I20260812 18:58:01.257130 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:58:01.257450 16547 external_mini_cluster.cc:1428] Setting key 40f17a940279bc6450f2ae75287f9a34
I20260812 18:58:01.259892 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--builtin_ntp_servers=127.16.40.212:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:58:01.260596 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.250889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:60169 (local address 127.16.40.254:35133)
0812 18:58:01.251049 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.251053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.252418 (+ 1365us) server_negotiation.cc:407] Connection header received
0812 18:58:01.254309 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.254313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.254375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.254464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:01.255705 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.256320 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.257923 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.258173 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.258313 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.259786 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:01.259794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:01.259800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:01.259817 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:01.259846 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:01.259859 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:01.259953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.260235 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.260332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:58:01.261651 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.251385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35133 (local address 127.16.40.193:60169)
0812 18:58:01.252236 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.252287 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.254002 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.254758 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.254766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.254875 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:01.255498 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.255524 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.256530 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.256533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.257755 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.257768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.258386 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.258391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.258613 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.259376 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:01.259378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:01.259429 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:01.259582 (+ 153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:01.259986 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:01.259997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:01.260004 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.260218 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.260638 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":142,"threads_started":1}
I20260812 18:58:01.263464 6079 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35133
I20260812 18:58:01.263784 6079 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:01.264462 6079 heartbeater.cc:511] Master 127.16.40.254:35133 requested a full tablet report, sending...
I20260812 18:58:01.266335 5850 ts_manager.cc:194] Registered new tserver with Master: 7735d92f8eff415cbb2185bc58e02874 (127.16.40.193:43585)
I20260812 18:58:01.267656 5850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.193:60169
W20260812 18:58:01.382704 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:01.383064 6082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:01.383143 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:01.387691 6082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:01.387801 6082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:01.387905 6082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:58:01.393139 6082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:01.394613 6082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:01.396330 6082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:01.407020 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:01.407022 6087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:01.407024 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:01.407361 6082 server_base.cc:1034] running on GCE node
I20260812 18:58:01.407936 6082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:01.408735 6082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:01.409933 6082 hybrid_clock.cc:648] HybridClock initialized: now 1786561081409905 us; error 50 us; skew 500 ppm
I20260812 18:58:01.411981 6082 webserver.cc:460] Webserver started at http://127.16.40.194:35581/ using document root <none> and password file <none>
I20260812 18:58:01.412707 6082 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:01.412808 6082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:01.413094 6082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:01.415500 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "42da43f8a8794e9a81d883c24b2ade72"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "b4acfd343013dc96e69b99d7c83abc5f"
server_key_iv: "47e27f6fc7ed41d826d6da4beb2ea1f9"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.416139 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "42da43f8a8794e9a81d883c24b2ade72"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "b4acfd343013dc96e69b99d7c83abc5f"
server_key_iv: "47e27f6fc7ed41d826d6da4beb2ea1f9"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.420390 6082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:01.423389 6096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.425061 6082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:01.425236 6082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "42da43f8a8794e9a81d883c24b2ade72"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "b4acfd343013dc96e69b99d7c83abc5f"
server_key_iv: "47e27f6fc7ed41d826d6da4beb2ea1f9"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.425379 6082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:01.451133 6082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:01.451974 6082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:01.452220 6082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:01.453020 6082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:01.454129 6082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:01.454205 6082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.454299 6082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:01.454355 6082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.464329 6082 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:43103
I20260812 18:58:01.464360 6241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:43103 every 8 connection(s)
I20260812 18:58:01.465432 6082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:58:01.469565 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6082
I20260812 18:58:01.469726 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:58:01.470055 16547 external_mini_cluster.cc:1428] Setting key 9e86d71e1a39f6bcccb1b3fde2109675
I20260812 18:58:01.472602 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--builtin_ntp_servers=127.16.40.212:42749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:58:01.475979 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.467473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:47623 (local address 127.16.40.254:35133)
0812 18:58:01.467611 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.467616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.468841 (+ 1225us) server_negotiation.cc:407] Connection header received
0812 18:58:01.470473 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.470478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.470545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.470622 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:01.471926 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.472459 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.473770 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.474013 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.474136 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.475424 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:01.475428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:01.475430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:01.475445 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:01.475465 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:01.475474 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:01.475568 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.475752 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.475823 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:58:01.477044 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.467850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35133 (local address 127.16.40.194:47623)
0812 18:58:01.468670 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.468718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.470238 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.470883 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.470892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.470993 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:01.471760 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.471782 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.472630 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.472633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.473634 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.473645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.474188 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.474192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.474388 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.475119 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:01.475121 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:01.475168 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:01.475243 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:01.475580 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:01.475592 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:01.475604 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.475738 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.476060 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260812 18:58:01.478380 6242 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35133
I20260812 18:58:01.478659 6242 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:01.479256 6242 heartbeater.cc:511] Master 127.16.40.254:35133 requested a full tablet report, sending...
I20260812 18:58:01.480495 5850 ts_manager.cc:194] Registered new tserver with Master: 42da43f8a8794e9a81d883c24b2ade72 (127.16.40.194:43103)
I20260812 18:58:01.481614 5850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.194:47623
W20260812 18:58:01.598037 6248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:01.598354 6248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:01.598394 6248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:01.602577 6248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:01.602645 6248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:01.602702 6248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:58:01.611210 6248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:42749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:35133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:01.612975 6248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:01.615058 6248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:01.628664 6255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:01.628798 6248 server_base.cc:1034] running on GCE node
W20260812 18:58:01.628629 6257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:01.629119 6254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:01.629724 6248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:01.630661 6248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:01.631893 6248 hybrid_clock.cc:648] HybridClock initialized: now 1786561081631849 us; error 50 us; skew 500 ppm
I20260812 18:58:01.634182 6248 webserver.cc:460] Webserver started at http://127.16.40.195:36989/ using document root <none> and password file <none>
I20260812 18:58:01.635083 6248 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:01.635205 6248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:01.635529 6248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:01.638375 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "687b2b0236f74770b6cd7a2f98aef70b"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "113fbfa3525d0cddaba03d08650f43a3"
server_key_iv: "8f8ed50c8458ad0da4d8cbc7b28c2197"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.639413 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "687b2b0236f74770b6cd7a2f98aef70b"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "113fbfa3525d0cddaba03d08650f43a3"
server_key_iv: "8f8ed50c8458ad0da4d8cbc7b28c2197"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.645126 6248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 18:58:01.649338 6266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.651221 6248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:01.651413 6248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "687b2b0236f74770b6cd7a2f98aef70b"
format_stamp: "Formatted at 2026-08-12 18:58:01 on dist-test-slave-rnrw"
server_key: "113fbfa3525d0cddaba03d08650f43a3"
server_key_iv: "8f8ed50c8458ad0da4d8cbc7b28c2197"
server_key_version: "encryptionkey@0"
I20260812 18:58:01.651602 6248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:01.682993 6248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:01.683722 6248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:01.683882 6248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:01.684450 6248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:01.685532 6248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:01.685596 6248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.685640 6248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:01.685655 6248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:01.695411 6248 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:35607
I20260812 18:58:01.695439 6401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:35607 every 8 connection(s)
I20260812 18:58:01.696542 6248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:58:01.701978 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6248
I20260812 18:58:01.702131 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:58:01.702425 16547 external_mini_cluster.cc:1428] Setting key 3b159589787726f7818a17224f256989
I20260812 18:58:01.707334 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.698750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:42365 (local address 127.16.40.254:35133)
0812 18:58:01.698898 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.698902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.700255 (+ 1353us) server_negotiation.cc:407] Connection header received
0812 18:58:01.701807 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.701810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.701878 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.701997 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:01.703067 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.703699 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.705174 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.705404 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.705521 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.706776 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:01.706779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:01.706781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:01.706795 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:01.706817 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:01.706825 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:01.706908 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.707103 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.707180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:58:01.708354 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.699205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:35133 (local address 127.16.40.195:42365)
0812 18:58:01.699976 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.700022 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.701572 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.702207 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.702219 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.702337 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:01.702913 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.702928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.703837 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.703840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.705009 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.705020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.705572 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.705576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.705763 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.706437 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:01.706439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:01.706487 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:01.706572 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:01.706924 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:01.706938 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:01.706948 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.707104 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.707472 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":161,"threads_started":1}
I20260812 18:58:01.709713 6402 heartbeater.cc:347] Connected to a master server at 127.16.40.254:35133
I20260812 18:58:01.709995 6402 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:01.710601 6402 heartbeater.cc:511] Master 127.16.40.254:35133 requested a full tablet report, sending...
I20260812 18:58:01.711822 5850 ts_manager.cc:194] Registered new tserver with Master: 687b2b0236f74770b6cd7a2f98aef70b (127.16.40.195:35607)
I20260812 18:58:01.712491 5850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.195:42365
I20260812 18:58:01.717581 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:58:01.724151 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.719914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.16.40.254:35133)
0812 18:58:01.720060 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.720064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.720138 (+ 74us) server_negotiation.cc:407] Connection header received
0812 18:58:01.720251 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.720253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.720297 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.720377 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:01.720757 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.721300 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.722362 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.722610 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.722701 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.723617 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:01.723621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:01.723623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:01.723638 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:01.723659 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:01.723667 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:01.723786 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.723977 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.724022 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 18:58:01.727746 5850 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:58:01.731197 5850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:58:01.746845 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.741933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.16.40.194:43103)
0812 18:58:01.742286 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.742302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.742324 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:58:01.742397 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.742403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.742580 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.742722 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:01.742967 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.743706 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.745613 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746154 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746283 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746502 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.746562 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.746664 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":132,"threads_started":1}
I20260812 18:58:01.746861 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.740585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:35607 (local address 127.0.0.1:32874)
0812 18:58:01.741098 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.741154 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.741299 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.742537 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.742539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.742556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:01.742658 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.742664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.744039 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.744043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.745425 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.745437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746441 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.746544 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746573 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.746628 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.746728 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":77,"thread_start_us":92,"threads_started":1}
I20260812 18:58:01.747152 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.741510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:43103 (local address 127.0.0.1:60442)
0812 18:58:01.741771 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.741784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.741872 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.742705 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.742708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.742728 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:01.742819 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.742829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.743827 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.743833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.745469 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.745479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746286 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.746394 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746430 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.746538 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.747045 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":264,"thread_start_us":89,"threads_started":1}
I20260812 18:58:01.747246 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.740867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.16.40.193:43585)
0812 18:58:01.741579 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.741587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.741619 (+ 32us) server_negotiation.cc:407] Connection header received
0812 18:58:01.741828 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.741835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.742071 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.742231 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:01.742678 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.743926 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.745394 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746220 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746343 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746552 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.746948 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.747071 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":119,"threads_started":1}
I20260812 18:58:01.747448 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.740754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.16.40.195:35607)
0812 18:58:01.741328 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.741979 (+ 651us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.742008 (+ 29us) server_negotiation.cc:407] Connection header received
0812 18:58:01.742174 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.742179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.742385 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.742530 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:01.742784 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.743913 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.745560 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746308 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746722 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746907 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.747164 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.747267 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":164,"threads_started":1}
I20260812 18:58:01.747535 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.741145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43585 (local address 127.0.0.1:58696)
0812 18:58:01.741544 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.741559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.741649 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.742337 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.742343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.742369 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:01.742539 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.742546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.744088 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.744090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.745260 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.745270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.746615 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.746621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.746712 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.746733 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.746797 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.746864 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":4,"thread_start_us":112,"threads_started":1}
I20260812 18:58:01.750352 5994 tablet_service.cc:1467] Processing CreateTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 (DEFAULT_TABLE table=test-table [id=530cc567b2f24412ba8d5491320839fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:01.750392 6156 tablet_service.cc:1467] Processing CreateTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 (DEFAULT_TABLE table=test-table [id=530cc567b2f24412ba8d5491320839fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:01.750396 6321 tablet_service.cc:1467] Processing CreateTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 (DEFAULT_TABLE table=test-table [id=530cc567b2f24412ba8d5491320839fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:01.751466 6321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:01.751466 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:01.751464 6156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:01.758176 6422 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: Bootstrap starting.
I20260812 18:58:01.759734 6423 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Bootstrap starting.
I20260812 18:58:01.761085 6422 tablet_bootstrap.cc:654] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:01.761816 6423 tablet_bootstrap.cc:654] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:01.761924 6422 log.cc:824] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:01.762578 6423 log.cc:824] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:01.763227 6424 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Bootstrap starting.
I20260812 18:58:01.764195 6422 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: No bootstrap required, opened a new log
I20260812 18:58:01.764424 6422 ts_tablet_manager.cc:1397] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:58:01.764618 6423 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: No bootstrap required, opened a new log
I20260812 18:58:01.764828 6423 ts_tablet_manager.cc:1397] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260812 18:58:01.765683 6424 tablet_bootstrap.cc:654] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:01.766683 6424 log.cc:824] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:01.768872 6422 raft_consensus.cc:348] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.769148 6422 raft_consensus.cc:374] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:01.769204 6422 raft_consensus.cc:729] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735d92f8eff415cbb2185bc58e02874, State: Initialized, Role: FOLLOWER
I20260812 18:58:01.769070 6423 raft_consensus.cc:348] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.769266 6424 tablet_bootstrap.cc:492] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: No bootstrap required, opened a new log
I20260812 18:58:01.769327 6423 raft_consensus.cc:374] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:01.769407 6423 raft_consensus.cc:729] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687b2b0236f74770b6cd7a2f98aef70b, State: Initialized, Role: FOLLOWER
I20260812 18:58:01.769498 6424 ts_tablet_manager.cc:1397] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 18:58:01.769770 6422 consensus_queue.cc:260] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.769870 6423 consensus_queue.cc:260] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.770526 6079 heartbeater.cc:503] Master 127.16.40.254:35133 was elected leader, sending a full tablet report...
I20260812 18:58:01.770923 6423 ts_tablet_manager.cc:1428] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:58:01.770936 6422 ts_tablet_manager.cc:1428] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 18:58:01.771364 6402 heartbeater.cc:503] Master 127.16.40.254:35133 was elected leader, sending a full tablet report...
I20260812 18:58:01.773723 6424 raft_consensus.cc:348] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.773999 6424 raft_consensus.cc:374] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:01.774078 6424 raft_consensus.cc:729] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42da43f8a8794e9a81d883c24b2ade72, State: Initialized, Role: FOLLOWER
I20260812 18:58:01.774510 6424 consensus_queue.cc:260] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.775311 6242 heartbeater.cc:503] Master 127.16.40.254:35133 was elected leader, sending a full tablet report...
I20260812 18:58:01.775689 6424 ts_tablet_manager.cc:1428] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 18:58:01.775718 6430 raft_consensus.cc:482] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:58:01.775873 6430 raft_consensus.cc:504] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.778642 6430 leader_election.cc:288] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 687b2b0236f74770b6cd7a2f98aef70b (127.16.40.195:35607), 7735d92f8eff415cbb2185bc58e02874 (127.16.40.193:43585)
I20260812 18:58:01.781683 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.777898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:46661 (local address 127.16.40.195:35607)
0812 18:58:01.778108 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.778115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.778131 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:58:01.778165 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.778167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.778202 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.778288 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:01.778555 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.779192 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.780719 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.781112 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.781239 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.781280 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.781482 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.781541 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 18:58:01.781668 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.777726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:35607 (local address 127.16.40.194:46661)
0812 18:58:01.777857 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.777876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.777961 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.778300 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.778303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.778320 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:01.778447 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.778454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.779324 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.779331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.780599 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.780609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.781250 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.781254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.781357 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.781384 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.781459 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.781528 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 18:58:01.782361 6344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35874f9a8e1543c7a4d3f19ddd3b4cb5" candidate_uuid: "42da43f8a8794e9a81d883c24b2ade72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687b2b0236f74770b6cd7a2f98aef70b" is_pre_election: true
I20260812 18:58:01.782742 6344 raft_consensus.cc:2454] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42da43f8a8794e9a81d883c24b2ade72 in term 0.
I20260812 18:58:01.783154 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.778122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43585 (local address 127.16.40.194:34737)
0812 18:58:01.778962 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.778977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:01.779123 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:01.779504 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:01.779506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:01.779522 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:01.779662 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.779668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.780742 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.780745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.781694 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:01.781705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.782704 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.782708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:01.782818 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.782843 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 18:58:01.782918 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 18:58:01.782988 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":118,"threads_started":1}
I20260812 18:58:01.783247 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.778474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:34737 (local address 127.16.40.193:43585)
0812 18:58:01.778600 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.778604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.779050 (+ 446us) server_negotiation.cc:407] Connection header received
0812 18:58:01.779248 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.779253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.779307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.779358 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:01.779858 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.780599 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.781848 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.782427 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.782538 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.782586 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.782988 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.783110 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:58:01.783363 6098 leader_election.cc:302] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42da43f8a8794e9a81d883c24b2ade72, 687b2b0236f74770b6cd7a2f98aef70b; no voters:
I20260812 18:58:01.783690 6430 raft_consensus.cc:2781] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:58:01.783804 6430 raft_consensus.cc:482] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:58:01.783872 6430 raft_consensus.cc:3037] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:01.783907 6024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35874f9a8e1543c7a4d3f19ddd3b4cb5" candidate_uuid: "42da43f8a8794e9a81d883c24b2ade72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735d92f8eff415cbb2185bc58e02874" is_pre_election: true
I20260812 18:58:01.784232 6024 raft_consensus.cc:2454] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42da43f8a8794e9a81d883c24b2ade72 in term 0.
I20260812 18:58:01.785245 6430 raft_consensus.cc:504] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.785676 6430 leader_election.cc:288] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [CANDIDATE]: Term 1 election: Requested vote from peers 687b2b0236f74770b6cd7a2f98aef70b (127.16.40.195:35607), 7735d92f8eff415cbb2185bc58e02874 (127.16.40.193:43585)
I20260812 18:58:01.786103 6344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35874f9a8e1543c7a4d3f19ddd3b4cb5" candidate_uuid: "42da43f8a8794e9a81d883c24b2ade72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687b2b0236f74770b6cd7a2f98aef70b"
I20260812 18:58:01.786229 6024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35874f9a8e1543c7a4d3f19ddd3b4cb5" candidate_uuid: "42da43f8a8794e9a81d883c24b2ade72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735d92f8eff415cbb2185bc58e02874"
I20260812 18:58:01.786275 6344 raft_consensus.cc:3037] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:01.786394 6024 raft_consensus.cc:3037] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:01.787478 6344 raft_consensus.cc:2454] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42da43f8a8794e9a81d883c24b2ade72 in term 1.
I20260812 18:58:01.787518 6024 raft_consensus.cc:2454] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42da43f8a8794e9a81d883c24b2ade72 in term 1.
I20260812 18:58:01.787881 6098 leader_election.cc:302] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42da43f8a8794e9a81d883c24b2ade72, 687b2b0236f74770b6cd7a2f98aef70b; no voters:
I20260812 18:58:01.788093 6430 raft_consensus.cc:2781] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:01.788362 6430 raft_consensus.cc:686] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 1 LEADER]: Becoming Leader. State: Replica: 42da43f8a8794e9a81d883c24b2ade72, State: Running, Role: LEADER
I20260812 18:58:01.788805 6430 consensus_queue.cc:237] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } }
I20260812 18:58:01.792125 5849 catalog_manager.cc:5495] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42da43f8a8794e9a81d883c24b2ade72 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "42da43f8a8794e9a81d883c24b2ade72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42da43f8a8794e9a81d883c24b2ade72" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 43103 } health_report { overall_health: HEALTHY } } }
I20260812 18:58:01.809221 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.805198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.16.40.194:43103)
0812 18:58:01.805336 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:01.805340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:01.805355 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:58:01.805571 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:01.805578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:01.805643 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:01.805754 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:01.806030 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.806589 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.807584 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:01.807811 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:01.807900 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:01.808191 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:01.808723 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:01.808846 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:01.808989 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 18:58:01.809028 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:58:01.815676 6024 raft_consensus.cc:1261] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 1 FOLLOWER]: Refusing update from remote peer 42da43f8a8794e9a81d883c24b2ade72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:01.815676 6344 raft_consensus.cc:1261] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 1 FOLLOWER]: Refusing update from remote peer 42da43f8a8794e9a81d883c24b2ade72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:01.816491 6432 consensus_queue.cc:1035] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7735d92f8eff415cbb2185bc58e02874" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:01.816860 6430 consensus_queue.cc:1035] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687b2b0236f74770b6cd7a2f98aef70b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 35607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:01.827648 6437 mvcc.cc:205] Tried to move back new op lower bound from 7317754191108853760 to 7317754191009882112. Current Snapshot: MvccSnapshot[applied={T|T < 7317754191108853760}]
I20260812 18:58:01.837672 5850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:01.837909 5850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:01.840734 5850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7613243322f9408c899a57844b807535: Sending DeleteTablet for 3 replicas of tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5
I20260812 18:58:01.841809 6321 tablet_service.cc:1514] Processing DeleteTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:01 UTC) from {username='slave'} at 127.0.0.1:32874
I20260812 18:58:01.841907 5994 tablet_service.cc:1514] Processing DeleteTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:01 UTC) from {username='slave'} at 127.0.0.1:58696
I20260812 18:58:01.841941 6156 tablet_service.cc:1514] Processing DeleteTablet for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:01 UTC) from {username='slave'} at 127.0.0.1:60442
I20260812 18:58:01.842499 6447 tablet_replica.cc:330] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874: stopping tablet replica
I20260812 18:58:01.842763 6445 tablet_replica.cc:330] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: stopping tablet replica
I20260812 18:58:01.843120 6448 tablet_replica.cc:330] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: stopping tablet replica
I20260812 18:58:01.843117 6445 raft_consensus.cc:2229] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:01.843317 6447 raft_consensus.cc:2229] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 7735d92f8eff415cbb2185bc58e02874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:01.843433 6448 raft_consensus.cc:2229] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:58:01.843585 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5925
I20260812 18:58:01.844159 6448 raft_consensus.cc:2258] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:01.847191 6448 ts_tablet_manager.cc:1905] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:01.852401 6448 ts_tablet_manager.cc:1918] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:01.852614 6448 log.cc:1196] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/35874f9a8e1543c7a4d3f19ddd3b4cb5
I20260812 18:58:01.853140 6448 ts_tablet_manager.cc:1939] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 42da43f8a8794e9a81d883c24b2ade72: Deleting consensus metadata
I20260812 18:58:01.854066 6445 raft_consensus.cc:2258] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:01.855144 5832 catalog_manager.cc:4841] TS 42da43f8a8794e9a81d883c24b2ade72 (127.16.40.194:43103): tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 (table test-table [id=530cc567b2f24412ba8d5491320839fb]) successfully deleted
I20260812 18:58:01.856937 6445 ts_tablet_manager.cc:1905] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:01.860189 6445 ts_tablet_manager.cc:1918] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:01.860421 6445 log.cc:1196] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/35874f9a8e1543c7a4d3f19ddd3b4cb5
I20260812 18:58:01.860980 6445 ts_tablet_manager.cc:1939] T 35874f9a8e1543c7a4d3f19ddd3b4cb5 P 687b2b0236f74770b6cd7a2f98aef70b: Deleting consensus metadata
I20260812 18:58:01.862684 5833 catalog_manager.cc:4841] TS 687b2b0236f74770b6cd7a2f98aef70b (127.16.40.195:35607): tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5 (table test-table [id=530cc567b2f24412ba8d5491320839fb]) successfully deleted
W20260812 18:58:01.868366 5832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:43585 (error 108)
I20260812 18:58:01.868616 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6082
I20260812 18:58:01.874612 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:01.872026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:43585 (local address 127.0.0.1:58698)
0812 18:58:01.874371 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:01.874481 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2210}
W20260812 18:58:01.874945 5832 catalog_manager.cc:4584] TS 7735d92f8eff415cbb2185bc58e02874 (127.16.40.193:43585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35874f9a8e1543c7a4d3f19ddd3b4cb5: Network error: Client connection negotiation failed: client connection to 127.16.40.193:43585: connect: Connection refused (error 111)
I20260812 18:58:01.880330 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6248
I20260812 18:58:01.890259 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 5812
2026-08-12T18:58:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1153 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:01 dist-test-slave-rnrw krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:01 dist-test-slave-rnrw krb5kdc[6453](info): set up 2 sockets
Aug 12 18:58:01 dist-test-slave-rnrw krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260812 18:58:03.037418 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:03.180821 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33843
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:03.302323 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:03.302656 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:03.302699 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:03.306945 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:03.307037 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:03.307057 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:03.307075 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:03.307091 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:03.312811 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:03.314066 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:03.315184 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:03.324721 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:03.324743 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:03.324828 6466 server_base.cc:1034] running on GCE node
W20260812 18:58:03.324731 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:03.325598 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:03.326889 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:03.328086 6466 hybrid_clock.cc:648] HybridClock initialized: now 1786561083328070 us; error 33 us; skew 500 ppm
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:03.331420 6466 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:03.332437 6466 webserver.cc:460] Webserver started at http://127.16.40.254:33393/ using document root <none> and password file <none>
I20260812 18:58:03.333091 6466 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:03.333161 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:03.333343 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:03.335525 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "da0bc244754c441a900c1e6d3d17cae8"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "2edb10ab66cdff5f8ad41dec1468d688"
server_key_iv: "ea4b9e5515f2e061da16d2dbc1bc922b"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.336118 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "da0bc244754c441a900c1e6d3d17cae8"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "2edb10ab66cdff5f8ad41dec1468d688"
server_key_iv: "ea4b9e5515f2e061da16d2dbc1bc922b"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.340703 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:03.343662 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.345395 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:03.345587 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "da0bc244754c441a900c1e6d3d17cae8"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "2edb10ab66cdff5f8ad41dec1468d688"
server_key_iv: "ea4b9e5515f2e061da16d2dbc1bc922b"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.345737 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:03.362416 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:03.365892 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:03.366087 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:03.373802 6466 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33843
I20260812 18:58:03.373831 6541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33843 every 8 connection(s)
I20260812 18:58:03.374904 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:03.378454 6542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:03.379453 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6466
I20260812 18:58:03.379606 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:03.379935 16547 external_mini_cluster.cc:1428] Setting key 04f13a814ce7d575a0fe37c63e42fca2
I20260812 18:58:03.385628 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: Bootstrap starting.
I20260812 18:58:03.388276 6542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:03.389214 6542 log.cc:824] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: Log is configured to *not* fsync() on all Append() calls
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:03.392088 6542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: No bootstrap required, opened a new log
I20260812 18:58:03.396148 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:03.381735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.16.40.254:33843)
0812 18:58:03.382265 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:03.382280 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:03.382320 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:58:03.382987 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:03.383005 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:03.383377 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:03.383765 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:03.384655 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.385738 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.386702 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.387224 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.387430 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:03.390502 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:03.390510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:03.390531 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:03.390569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:03.392657 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.393126 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.393133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.393140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.393297 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.393596 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.393599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.393601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.394174 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:03.394411 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:03.394649 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 18:58:03.395018 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":152,"threads_started":1}
I20260812 18:58:03.397116 6542 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } }
I20260812 18:58:03.397421 6542 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:03.397485 6542 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0bc244754c441a900c1e6d3d17cae8, State: Initialized, Role: FOLLOWER
I20260812 18:58:03.397933 6542 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } }
I20260812 18:58:03.398063 6542 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:03.398162 6542 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:03.398293 6542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:03.399501 6542 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } }
I20260812 18:58:03.399878 6542 leader_election.cc:302] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0bc244754c441a900c1e6d3d17cae8; no voters:
I20260812 18:58:03.400223 6542 leader_election.cc:288] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:03.400373 6548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:03.400657 6548 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [term 1 LEADER]: Becoming Leader. State: Replica: da0bc244754c441a900c1e6d3d17cae8, State: Running, Role: LEADER
I20260812 18:58:03.401016 6548 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } }
I20260812 18:58:03.401726 6542 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:03.402968 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0bc244754c441a900c1e6d3d17cae8. Latest consensus state: current_term: 1 leader_uuid: "da0bc244754c441a900c1e6d3d17cae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } } }
I20260812 18:58:03.403007 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0bc244754c441a900c1e6d3d17cae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bc244754c441a900c1e6d3d17cae8" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33843 } } }
I20260812 18:58:03.403118 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:03.403129 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:03.403556 6560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:03.407146 6560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:03.413241 6560 catalog_manager.cc:1347] Generated new cluster ID: 66999fa920a546a7a2ed16fad964ae53
I20260812 18:58:03.413340 6560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:03.426528 6560 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:03.427953 6560 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:03.441522 6560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: Generated new TSK 0
I20260812 18:58:03.442534 6560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.40.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
Entry for principal kudu/127.16.40.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab.
I20260812 18:58:03.484457 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:03.607862 6574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:03.608233 6574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:03.608321 6574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:03.612578 6574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:03.612695 6574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:03.612797 6574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:58:03.618707 6574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.193.keytab
--principal=kudu/127.16.40.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:03.619967 6574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:03.621145 6574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:03.631917 6582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:03.632061 6574 server_base.cc:1034] running on GCE node
W20260812 18:58:03.631899 6581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:03.631906 6584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:03.632826 6574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:03.633623 6574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:03.634838 6574 hybrid_clock.cc:648] HybridClock initialized: now 1786561083634792 us; error 59 us; skew 500 ppm
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:03.638047 6574 init.cc:377] Logged in from keytab as kudu/127.16.40.193@KRBTEST.COM (short username kudu)
I20260812 18:58:03.639065 6574 webserver.cc:460] Webserver started at http://127.16.40.193:38009/ using document root <none> and password file <none>
I20260812 18:58:03.639693 6574 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:03.639791 6574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:03.640054 6574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:03.642323 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "f50f170c331c4dc3968fea1092e124c4"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "a3d7de98549ca8ac980894d86fb0ffd7"
server_key_iv: "30f05e1ebcf5b2470b553f38e95e920f"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.642916 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "f50f170c331c4dc3968fea1092e124c4"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "a3d7de98549ca8ac980894d86fb0ffd7"
server_key_iv: "30f05e1ebcf5b2470b553f38e95e920f"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.646894 6574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 18:58:03.649912 6594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.651503 6574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:03.651676 6574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "f50f170c331c4dc3968fea1092e124c4"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "a3d7de98549ca8ac980894d86fb0ffd7"
server_key_iv: "30f05e1ebcf5b2470b553f38e95e920f"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.651813 6574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:03.675737 6574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:03.679608 6574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:03.679859 6574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:03.680622 6574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:03.681552 6574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:03.681613 6574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.681687 6574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:03.681720 6574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.691187 6574 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:37417
I20260812 18:58:03.691208 6729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:37417 every 8 connection(s)
I20260812 18:58:03.692185 6574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:58:03.693612 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6574
I20260812 18:58:03.693756 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:58:03.694104 16547 external_mini_cluster.cc:1428] Setting key 89fdf4b27eb68286b222bef2459ad5fd
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.193@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.194@KRBTEST.COM" created.
I20260812 18:58:03.708727 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:03.694548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:40639 (local address 127.16.40.254:33843)
0812 18:58:03.694721 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:03.694726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:03.695832 (+ 1106us) server_negotiation.cc:407] Connection header received
0812 18:58:03.697330 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:03.697334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:03.697392 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:03.697484 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:03.698718 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.699499 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.700864 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.701136 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.701296 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:03.704393 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:03.704400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:03.704404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:03.704443 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:03.706067 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.706660 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.706662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.706664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.706747 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.707131 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.707135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.707136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.707365 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:03.707537 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:03.708188 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 18:58:03.708373 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:58:03.709785 6733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:03.695047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33843 (local address 127.16.40.193:40639)
0812 18:58:03.695648 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:03.695716 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:03.697109 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:03.697660 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:03.697668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:03.698054 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:03.698565 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:03.698580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.699660 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.699663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:03.700717 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:03.700730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.701355 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.701359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:03.701628 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:03.702343 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:03.702372 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:03.704165 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:03.706219 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:03.706230 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:03.706257 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:03.706550 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:03.706839 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:03.706842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:03.706843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:03.706998 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:03.707541 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:03.707547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:03.707859 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 18:58:03.708100 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 18:58:03.708462 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":137,"threads_started":1}
I20260812 18:58:03.711684 6730 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33843
I20260812 18:58:03.712008 6730 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:03.712740 6730 heartbeater.cc:511] Master 127.16.40.254:33843 requested a full tablet report, sending...
I20260812 18:58:03.714814 6502 ts_manager.cc:194] Registered new tserver with Master: f50f170c331c4dc3968fea1092e124c4 (127.16.40.193:37417)
I20260812 18:58:03.716747 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.193@KRBTEST.COM'} at 127.16.40.193:40639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
Entry for principal kudu/127.16.40.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab.
I20260812 18:58:03.729097 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:03.850960 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:03.851282 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:03.851351 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:03.855711 6739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:03.855805 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:03.855894 6739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:58:03.861644 6739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.194.keytab
--principal=kudu/127.16.40.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:03.862835 6739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:03.863906 6739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:03.874102 6745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:03.874147 6747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:03.874135 6744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:03.874190 6739 server_base.cc:1034] running on GCE node
I20260812 18:58:03.874861 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:03.875567 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:03.876729 6739 hybrid_clock.cc:648] HybridClock initialized: now 1786561083876714 us; error 28 us; skew 500 ppm
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:03.879796 6739 init.cc:377] Logged in from keytab as kudu/127.16.40.194@KRBTEST.COM (short username kudu)
I20260812 18:58:03.880875 6739 webserver.cc:460] Webserver started at http://127.16.40.194:39569/ using document root <none> and password file <none>
I20260812 18:58:03.881515 6739 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:03.881624 6739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:03.881879 6739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:03.884130 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "a9f9e63921b04d729334ef54edf71d09"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "c32adcc5916480bfd46006e17aa95ad5"
server_key_iv: "36db8a5bfe9c61a81438ca6cdf2f013a"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.884800 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "a9f9e63921b04d729334ef54edf71d09"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "c32adcc5916480bfd46006e17aa95ad5"
server_key_iv: "36db8a5bfe9c61a81438ca6cdf2f013a"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.889103 6739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:58:03.891824 6754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.893218 6739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:03.893375 6739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "a9f9e63921b04d729334ef54edf71d09"
format_stamp: "Formatted at 2026-08-12 18:58:03 on dist-test-slave-rnrw"
server_key: "c32adcc5916480bfd46006e17aa95ad5"
server_key_iv: "36db8a5bfe9c61a81438ca6cdf2f013a"
server_key_version: "encryptionkey@0"
I20260812 18:58:03.893508 6739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:03.910147 6739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:03.914180 6739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:03.914415 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:03.915118 6739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:03.916110 6739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:03.916184 6739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.916251 6739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:03.916302 6739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:03.926003 6739 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:37837
I20260812 18:58:03.926023 6883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:37837 every 8 connection(s)
I20260812 18:58:03.926993 6739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:58:03.927737 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6739
I20260812 18:58:03.927881 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:58:03.928184 16547 external_mini_cluster.cc:1428] Setting key e900f6efbb4eaa95fe4a2ccb508370ff
Aug 12 18:58:03 dist-test-slave-rnrw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561083, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.194@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.195@KRBTEST.COM; defaulting to no policy
I20260812 18:58:03.943985 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:03.930130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:38877 (local address 127.16.40.254:33843)
0812 18:58:03.930263 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:03.930266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:03.931402 (+ 1136us) server_negotiation.cc:407] Connection header received
0812 18:58:03.933362 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:03.933366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:03.933424 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:03.933508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:03.934731 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.935571 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.936862 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.937244 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.937374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:03.939959 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:03.939966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:03.939971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:03.940004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:03.941688 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.942211 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.942215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.942218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.942313 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:03.942704 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:03.942706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:03.942708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:03.942912 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:03.942995 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:03.943695 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 18:58:03.943808 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.195@KRBTEST.COM" created.
I20260812 18:58:03.945107 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:03.930700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33843 (local address 127.16.40.194:38877)
0812 18:58:03.931249 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:03.931295 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:03.933139 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:03.933706 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:03.933713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:03.934085 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:03.934581 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:03.934594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.935723 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.935726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:03.936734 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:03.936745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:03.937408 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:03.937411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:03.937554 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:03.938252 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:03.938269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:03.939767 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:03.941814 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:03.941820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:03.941834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:03.942106 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:03.942427 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:03.942430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:03.942432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:03.942602 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:03.943041 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:03.943046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:03.943363 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 18:58:03.943622 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 18:58:03.943916 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":157,"threads_started":1}
I20260812 18:58:03.946439 6884 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33843
I20260812 18:58:03.946739 6884 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:03.947424 6884 heartbeater.cc:511] Master 127.16.40.254:33843 requested a full tablet report, sending...
I20260812 18:58:03.948702 6502 ts_manager.cc:194] Registered new tserver with Master: a9f9e63921b04d729334ef54edf71d09 (127.16.40.194:37837)
I20260812 18:58:03.949322 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.194@KRBTEST.COM'} at 127.16.40.194:38877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
Entry for principal kudu/127.16.40.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab.
I20260812 18:58:03.963912 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:04.083442 6892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:04.083791 6892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:04.083871 6892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:04.088025 6892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:04.088141 6892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:04.088243 6892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:58:04.093947 6892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.195.keytab
--principal=kudu/127.16.40.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:33843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:04.095196 6892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:04.096315 6892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:04.107234 6897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:04.107373 6898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:04.107431 6892 server_base.cc:1034] running on GCE node
W20260812 18:58:04.107365 6900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:04.108114 6892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:04.108891 6892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:04.110086 6892 hybrid_clock.cc:648] HybridClock initialized: now 1786561084110060 us; error 37 us; skew 500 ppm
Aug 12 18:58:04 dist-test-slave-rnrw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:04.113617 6892 init.cc:377] Logged in from keytab as kudu/127.16.40.195@KRBTEST.COM (short username kudu)
I20260812 18:58:04.114583 6892 webserver.cc:460] Webserver started at http://127.16.40.195:41819/ using document root <none> and password file <none>
I20260812 18:58:04.115149 6892 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:04.115211 6892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:04.115386 6892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:04.117733 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "04df890fdc6d4394b781cc9998b7602b"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "8d795a4b2d8d3e783213f971dabbcfb9"
server_key_iv: "2a20c83b68f3b41f8de3d7a89ece8842"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.118333 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "04df890fdc6d4394b781cc9998b7602b"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "8d795a4b2d8d3e783213f971dabbcfb9"
server_key_iv: "2a20c83b68f3b41f8de3d7a89ece8842"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.122785 6892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:04.126202 6909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:04.127902 6892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:58:04.128139 6892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "04df890fdc6d4394b781cc9998b7602b"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "8d795a4b2d8d3e783213f971dabbcfb9"
server_key_iv: "2a20c83b68f3b41f8de3d7a89ece8842"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.128295 6892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:04.166302 6892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:04.170346 6892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:04.170640 6892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:04.171341 6892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:04.172318 6892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:04.172391 6892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:04.172469 6892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:04.172514 6892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:04.182735 6892 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:38715
I20260812 18:58:04.182785 7047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:38715 every 8 connection(s)
I20260812 18:58:04.183777 6892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:58:04.193410 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 6892
I20260812 18:58:04.193557 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:58:04.193936 16547 external_mini_cluster.cc:1428] Setting key a753706107a714521839d35bf091e593
Aug 12 18:58:04 dist-test-slave-rnrw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561084, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.195@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:04.199486 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.186012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:46145 (local address 127.16.40.254:33843)
0812 18:58:04.186218 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.186222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.187093 (+ 871us) server_negotiation.cc:407] Connection header received
0812 18:58:04.188502 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.188505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.188614 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.188709 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:04.190012 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.190556 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.191867 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.192095 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.192185 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.195522 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:04.195527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:04.195529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:04.195569 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:04.197251 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:04.197857 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:04.197860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:04.197861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:04.197941 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:04.198324 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:04.198326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:04.198327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:04.198577 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:04.198672 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.199241 (+ 569us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.199359 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 18:58:04.200397 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.186390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:33843 (local address 127.16.40.195:46145)
0812 18:58:04.186909 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.186956 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.188281 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.188894 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.188902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.189317 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:04.189844 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.189858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.190704 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.190708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.191726 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.191738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.192282 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.192285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.192456 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.193185 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:04.193210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:04.195338 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:04.197428 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:04.197442 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:04.197459 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:04.197766 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:04.198066 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:04.198069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:04.198071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:04.198233 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:04.198713 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:04.198719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:04.199001 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.199180 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.199469 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":139,"threads_started":1}
I20260812 18:58:04.201859 7048 heartbeater.cc:347] Connected to a master server at 127.16.40.254:33843
I20260812 18:58:04.202153 7048 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:04.202808 7048 heartbeater.cc:511] Master 127.16.40.254:33843 requested a full tablet report, sending...
I20260812 18:58:04.204080 6502 ts_manager.cc:194] Registered new tserver with Master: 04df890fdc6d4394b781cc9998b7602b (127.16.40.195:38715)
I20260812 18:58:04.204794 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.40.195@KRBTEST.COM'} at 127.16.40.195:46145
I20260812 18:58:04.208835 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:58:04.219349 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.211109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.16.40.254:33843)
0812 18:58:04.211296 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.211301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.211500 (+ 199us) server_negotiation.cc:407] Connection header received
0812 18:58:04.211694 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.211697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.211740 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.211828 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:04.212648 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.213169 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.214049 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.214283 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.214400 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.215866 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:04.215873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:04.215880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:04.215917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:04.217463 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:04.217928 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:04.217932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:04.217934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:04.218027 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:04.218322 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:04.218326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:04.218328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:04.218630 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:04.218782 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.219047 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.219173 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 18:58:04.222883 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:58:04.226181 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:58:04.243022 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.237183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38715 (local address 127.0.0.1:44650)
0812 18:58:04.238091 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.238106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.238232 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.238736 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.238739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.238759 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:04.239005 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.239016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240268 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.240273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.241502 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.241513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.242393 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.242396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.242508 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.242548 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.242712 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.242864 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":88,"thread_start_us":113,"threads_started":1}
I20260812 18:58:04.243031 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.237630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.16.40.195:38715)
0812 18:58:04.238123 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.238141 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.238172 (+ 31us) server_negotiation.cc:407] Connection header received
0812 18:58:04.238406 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.238412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.238572 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.238742 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:04.239161 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240096 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.241638 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.242248 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.242391 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.242576 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.242750 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.242864 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":170,"threads_started":1}
I20260812 18:58:04.244493 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.236292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:37837 (local address 127.0.0.1:43160)
0812 18:58:04.237576 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.237644 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.237814 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.238700 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.238703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.238716 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:04.239047 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.239059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240769 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.240773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.242456 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.242470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.243986 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.243990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.244118 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.244146 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.244223 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.244309 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":295,"thread_start_us":117,"threads_started":1}
I20260812 18:58:04.244983 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.236570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37417 (local address 127.0.0.1:51560)
0812 18:58:04.237576 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.237644 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.237806 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.238623 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.238633 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.238664 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:04.239079 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.239090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240769 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.240773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.242447 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.242461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.244363 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.244367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.244457 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.244483 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.244744 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.244835 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":361,"thread_start_us":128,"threads_started":1}
I20260812 18:58:04.245115 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.236532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.16.40.194:37837)
0812 18:58:04.237272 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.237288 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.237746 (+ 458us) server_negotiation.cc:407] Connection header received
0812 18:58:04.238009 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.238016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.238260 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.238400 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:04.239220 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240607 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.242618 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.243614 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.243710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.243861 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.244822 (+ 961us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.244940 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":136,"threads_started":1}
I20260812 18:58:04.245591 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.236919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.16.40.193:37417)
0812 18:58:04.237365 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.237374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.237987 (+ 613us) server_negotiation.cc:407] Connection header received
0812 18:58:04.238069 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.238074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.238260 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.238405 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:04.239230 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.240603 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.242617 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.243999 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.244127 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.244295 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.245310 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.245437 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":128,"threads_started":1}
I20260812 18:58:04.246877 6969 tablet_service.cc:1467] Processing CreateTablet for tablet 28611dc08a244952828570dc1ab9c2b4 (DEFAULT_TABLE table=test-table [id=29075a3531ac4fac977d979cbb9b3853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:04.248224 6969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28611dc08a244952828570dc1ab9c2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:04.250631 6648 tablet_service.cc:1467] Processing CreateTablet for tablet 28611dc08a244952828570dc1ab9c2b4 (DEFAULT_TABLE table=test-table [id=29075a3531ac4fac977d979cbb9b3853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:04.251788 6648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28611dc08a244952828570dc1ab9c2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:04.248073 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 28611dc08a244952828570dc1ab9c2b4 (DEFAULT_TABLE table=test-table [id=29075a3531ac4fac977d979cbb9b3853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:04.253461 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28611dc08a244952828570dc1ab9c2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:04.256273 7066 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Bootstrap starting.
I20260812 18:58:04.259356 7066 tablet_bootstrap.cc:654] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:04.260941 7067 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Bootstrap starting.
I20260812 18:58:04.260947 7069 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Bootstrap starting.
I20260812 18:58:04.261334 7066 log.cc:824] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:04.263274 7067 tablet_bootstrap.cc:654] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:04.263454 7069 tablet_bootstrap.cc:654] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:04.264113 7067 log.cc:824] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:04.264173 7066 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: No bootstrap required, opened a new log
I20260812 18:58:04.264415 7066 ts_tablet_manager.cc:1397] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 18:58:04.264981 7069 log.cc:824] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:04.266551 7067 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: No bootstrap required, opened a new log
I20260812 18:58:04.266774 7067 ts_tablet_manager.cc:1397] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:58:04.267289 7069 tablet_bootstrap.cc:492] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: No bootstrap required, opened a new log
I20260812 18:58:04.267635 7069 ts_tablet_manager.cc:1397] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:58:04.269173 7066 raft_consensus.cc:348] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.269522 7066 raft_consensus.cc:374] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:04.269591 7066 raft_consensus.cc:729] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04df890fdc6d4394b781cc9998b7602b, State: Initialized, Role: FOLLOWER
I20260812 18:58:04.270162 7066 consensus_queue.cc:260] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.270714 7067 raft_consensus.cc:348] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.271013 7048 heartbeater.cc:503] Master 127.16.40.254:33843 was elected leader, sending a full tablet report...
I20260812 18:58:04.271027 7067 raft_consensus.cc:374] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:04.271100 7067 raft_consensus.cc:729] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f170c331c4dc3968fea1092e124c4, State: Initialized, Role: FOLLOWER
I20260812 18:58:04.271413 7066 ts_tablet_manager.cc:1428] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:58:04.271618 7067 consensus_queue.cc:260] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.272297 6730 heartbeater.cc:503] Master 127.16.40.254:33843 was elected leader, sending a full tablet report...
I20260812 18:58:04.272679 7067 ts_tablet_manager.cc:1428] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260812 18:58:04.273398 7069 raft_consensus.cc:348] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.273701 7069 raft_consensus.cc:374] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:04.273780 7069 raft_consensus.cc:729] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f9e63921b04d729334ef54edf71d09, State: Initialized, Role: FOLLOWER
I20260812 18:58:04.274307 7069 consensus_queue.cc:260] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.275511 7069 ts_tablet_manager.cc:1428] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 18:58:04.275969 6884 heartbeater.cc:503] Master 127.16.40.254:33843 was elected leader, sending a full tablet report...
W20260812 18:58:04.428272 6885 tablet.cc:2369] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:58:04.434840 7049 tablet.cc:2369] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 18:58:04.443913 6731 tablet.cc:2369] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:58:04.542371 7076 raft_consensus.cc:482] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:58:04.542698 7076 raft_consensus.cc:504] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.544014 7076 leader_election.cc:288] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50f170c331c4dc3968fea1092e124c4 (127.16.40.193:37417), 04df890fdc6d4394b781cc9998b7602b (127.16.40.195:38715)
I20260812 18:58:04.550235 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.544468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:54257 (local address 127.16.40.195:38715)
0812 18:58:04.544641 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.544648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.545042 (+ 394us) server_negotiation.cc:407] Connection header received
0812 18:58:04.545113 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.545116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.545165 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.545254 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:04.545781 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.546725 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.548764 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.549431 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.549578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.549640 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.549991 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.550083 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:58:04.550235 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.544481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:42995 (local address 127.16.40.193:37417)
0812 18:58:04.544666 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.544671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.544946 (+ 275us) server_negotiation.cc:407] Connection header received
0812 18:58:04.545085 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.545089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.545149 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.545253 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:04.545781 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.546696 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.548764 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.549431 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.549578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.549640 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.549997 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.550083 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 18:58:04.550313 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.544348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:38715 (local address 127.16.40.194:54257)
0812 18:58:04.544914 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.544928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.545025 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.545300 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.545304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.545349 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:04.545605 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.545611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.546863 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.546867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.548585 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.548600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.549585 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.549590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.549829 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.549865 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.549952 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.550101 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":387,"spinlock_wait_cycles":34944,"thread_start_us":106,"threads_started":1}
I20260812 18:58:04.550312 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.544348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37417 (local address 127.16.40.194:42995)
0812 18:58:04.544801 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.544816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:04.544952 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:04.545300 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:04.545304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:04.545350 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:04.545598 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.545607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.546863 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.546867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.548587 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:04.548600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.549608 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.549611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:04.549834 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.549878 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 18:58:04.549951 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 18:58:04.550101 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":376,"spinlock_wait_cycles":8576,"thread_start_us":105,"threads_started":1}
I20260812 18:58:04.551260 6991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28611dc08a244952828570dc1ab9c2b4" candidate_uuid: "a9f9e63921b04d729334ef54edf71d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04df890fdc6d4394b781cc9998b7602b" is_pre_election: true
I20260812 18:58:04.551261 6673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28611dc08a244952828570dc1ab9c2b4" candidate_uuid: "a9f9e63921b04d729334ef54edf71d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f170c331c4dc3968fea1092e124c4" is_pre_election: true
I20260812 18:58:04.551607 6991 raft_consensus.cc:2454] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f9e63921b04d729334ef54edf71d09 in term 0.
I20260812 18:58:04.551657 6673 raft_consensus.cc:2454] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9f9e63921b04d729334ef54edf71d09 in term 0.
I20260812 18:58:04.552173 6756 leader_election.cc:302] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04df890fdc6d4394b781cc9998b7602b, a9f9e63921b04d729334ef54edf71d09, f50f170c331c4dc3968fea1092e124c4; no voters:
I20260812 18:58:04.552496 7076 raft_consensus.cc:2781] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:58:04.552623 7076 raft_consensus.cc:482] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:58:04.552681 7076 raft_consensus.cc:3037] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:04.553807 7076 raft_consensus.cc:504] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.554222 7076 leader_election.cc:288] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [CANDIDATE]: Term 1 election: Requested vote from peers f50f170c331c4dc3968fea1092e124c4 (127.16.40.193:37417), 04df890fdc6d4394b781cc9998b7602b (127.16.40.195:38715)
I20260812 18:58:04.554631 6673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28611dc08a244952828570dc1ab9c2b4" candidate_uuid: "a9f9e63921b04d729334ef54edf71d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f170c331c4dc3968fea1092e124c4"
I20260812 18:58:04.554749 6673 raft_consensus.cc:3037] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:04.554720 6991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28611dc08a244952828570dc1ab9c2b4" candidate_uuid: "a9f9e63921b04d729334ef54edf71d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04df890fdc6d4394b781cc9998b7602b"
I20260812 18:58:04.554885 6991 raft_consensus.cc:3037] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:04.555841 6673 raft_consensus.cc:2454] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f9e63921b04d729334ef54edf71d09 in term 1.
I20260812 18:58:04.556195 6991 raft_consensus.cc:2454] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9f9e63921b04d729334ef54edf71d09 in term 1.
I20260812 18:58:04.556200 6756 leader_election.cc:302] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f9e63921b04d729334ef54edf71d09, f50f170c331c4dc3968fea1092e124c4; no voters:
I20260812 18:58:04.556402 7076 raft_consensus.cc:2781] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:04.556710 7076 raft_consensus.cc:686] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 1 LEADER]: Becoming Leader. State: Replica: a9f9e63921b04d729334ef54edf71d09, State: Running, Role: LEADER
I20260812 18:58:04.557117 7076 consensus_queue.cc:237] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } }
I20260812 18:58:04.560478 6502 catalog_manager.cc:5495] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9f9e63921b04d729334ef54edf71d09 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "a9f9e63921b04d729334ef54edf71d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f9e63921b04d729334ef54edf71d09" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 37837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 } health_report { overall_health: UNKNOWN } } }
I20260812 18:58:04.611943 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.607774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.16.40.194:37837)
0812 18:58:04.607944 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.607947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.607963 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:58:04.608056 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.608059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.608120 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.608220 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:04.608574 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.609279 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.610194 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.610503 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.610593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.610930 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:04.611394 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:04.611508 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.611689 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.611752 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 18:58:04.618453 6991 raft_consensus.cc:1261] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 1 FOLLOWER]: Refusing update from remote peer a9f9e63921b04d729334ef54edf71d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:04.618453 6673 raft_consensus.cc:1261] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 1 FOLLOWER]: Refusing update from remote peer a9f9e63921b04d729334ef54edf71d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:04.619230 7079 consensus_queue.cc:1035] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50f170c331c4dc3968fea1092e124c4" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:04.619468 7076 consensus_queue.cc:1035] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04df890fdc6d4394b781cc9998b7602b" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 38715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:04.628104 7084 mvcc.cc:205] Tried to move back new op lower bound from 7317754202589102080 to 7317754202348945408. Current Snapshot: MvccSnapshot[applied={T|T < 7317754202589102080}]
I20260812 18:58:04.629029 7086 mvcc.cc:205] Tried to move back new op lower bound from 7317754202589102080 to 7317754202348945408. Current Snapshot: MvccSnapshot[applied={T|T < 7317754202589102080}]
I20260812 18:58:04.638535 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:04.638759 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:04.641633 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da0bc244754c441a900c1e6d3d17cae8: Sending DeleteTablet for 3 replicas of tablet 28611dc08a244952828570dc1ab9c2b4
I20260812 18:58:04.642576 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 28611dc08a244952828570dc1ab9c2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:04 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:43160
I20260812 18:58:04.642692 6648 tablet_service.cc:1514] Processing DeleteTablet for tablet 28611dc08a244952828570dc1ab9c2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:04 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:51560
I20260812 18:58:04.642699 6969 tablet_service.cc:1514] Processing DeleteTablet for tablet 28611dc08a244952828570dc1ab9c2b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:04 UTC) from {username='kudu', principal='kudu/127.16.40.254@KRBTEST.COM'} at 127.0.0.1:44650
I20260812 18:58:04.643290 7095 tablet_replica.cc:330] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4: stopping tablet replica
I20260812 18:58:04.643355 7097 tablet_replica.cc:330] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: stopping tablet replica
I20260812 18:58:04.643738 7095 raft_consensus.cc:2229] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:04.643740 7097 raft_consensus.cc:2229] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:04.644114 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6574
I20260812 18:58:04.644122 7097 raft_consensus.cc:2258] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:04.644160 7095 raft_consensus.cc:2258] T 28611dc08a244952828570dc1ab9c2b4 P f50f170c331c4dc3968fea1092e124c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:04.644528 7094 tablet_replica.cc:330] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: stopping tablet replica
I20260812 18:58:04.644869 7094 raft_consensus.cc:2229] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:58:04.645318 7094 raft_consensus.cc:2258] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:04.646636 7097 ts_tablet_manager.cc:1905] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:04.651357 7097 ts_tablet_manager.cc:1918] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:04.651820 7097 log.cc:1196] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/28611dc08a244952828570dc1ab9c2b4
I20260812 18:58:04.651968 7094 ts_tablet_manager.cc:1905] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:04.652491 7097 ts_tablet_manager.cc:1939] T 28611dc08a244952828570dc1ab9c2b4 P 04df890fdc6d4394b781cc9998b7602b: Deleting consensus metadata
I20260812 18:58:04.654569 6487 catalog_manager.cc:4841] TS 04df890fdc6d4394b781cc9998b7602b (127.16.40.195:38715): tablet 28611dc08a244952828570dc1ab9c2b4 (table test-table [id=29075a3531ac4fac977d979cbb9b3853]) successfully deleted
I20260812 18:58:04.654838 7094 ts_tablet_manager.cc:1918] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:04.654954 7094 log.cc:1196] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/28611dc08a244952828570dc1ab9c2b4
I20260812 18:58:04.655278 7094 ts_tablet_manager.cc:1939] T 28611dc08a244952828570dc1ab9c2b4 P a9f9e63921b04d729334ef54edf71d09: Deleting consensus metadata
I20260812 18:58:04.656695 6487 catalog_manager.cc:4841] TS a9f9e63921b04d729334ef54edf71d09 (127.16.40.194:37837): tablet 28611dc08a244952828570dc1ab9c2b4 (table test-table [id=29075a3531ac4fac977d979cbb9b3853]) successfully deleted
W20260812 18:58:04.665127 6485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:37417 (error 108)
I20260812 18:58:04.665361 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6739
I20260812 18:58:04.669178 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.668809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37417 (local address 127.0.0.1:51574)
0812 18:58:04.668961 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:04.669055 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:37417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 18:58:04.669554 6485 catalog_manager.cc:4584] TS f50f170c331c4dc3968fea1092e124c4 (127.16.40.193:37417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28611dc08a244952828570dc1ab9c2b4: Network error: Client connection negotiation failed: client connection to 127.16.40.193:37417: connect: Connection refused (error 111)
I20260812 18:58:04.675858 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6892
I20260812 18:58:04.685791 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 6466
2026-08-12T18:58:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T18:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:04Z Disabled control of system clock
I20260812 18:58:04.724407 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:34851
--webserver_interface=127.16.40.254
--webserver_port=0
--builtin_ntp_servers=127.16.40.212:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:58:04.845891 7103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:04.846235 7103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:04.846273 7103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:04.850490 7103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:04.850565 7103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:04.850581 7103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:04.850600 7103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:04.850615 7103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:04.855655 7103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:34851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:04.856880 7103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:04.858310 7103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:58:04.868243 7103 server_base.cc:1034] running on GCE node
W20260812 18:58:04.868183 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:04.868181 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:04.868178 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:04.869143 7103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:04.870569 7103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:04.871797 7103 hybrid_clock.cc:648] HybridClock initialized: now 1786561084871784 us; error 59 us; skew 500 ppm
I20260812 18:58:04.874296 7103 webserver.cc:460] Webserver started at http://127.16.40.254:32829/ using document root <none> and password file <none>
I20260812 18:58:04.875051 7103 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:04.875173 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:04.875470 7103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:04.878093 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "453261adcf684e729eb3ad5eddf7e46a"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "b1e651d558e38dd9133267e7de490b46"
server_key_iv: "8d52e8db687bc2a32871d20b3290923c"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.878813 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "453261adcf684e729eb3ad5eddf7e46a"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "b1e651d558e38dd9133267e7de490b46"
server_key_iv: "8d52e8db687bc2a32871d20b3290923c"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.883615 7103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:04.886656 7123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:04.888164 7103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:58:04.888350 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "453261adcf684e729eb3ad5eddf7e46a"
format_stamp: "Formatted at 2026-08-12 18:58:04 on dist-test-slave-rnrw"
server_key: "b1e651d558e38dd9133267e7de490b46"
server_key_iv: "8d52e8db687bc2a32871d20b3290923c"
server_key_version: "encryptionkey@0"
I20260812 18:58:04.888495 7103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:04.912462 7103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:04.913487 7103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:04.913754 7103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:04.922514 7103 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:34851
I20260812 18:58:04.922689 7179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:34851 every 8 connection(s)
I20260812 18:58:04.923624 7103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:04.927474 7181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:04.932292 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7103
I20260812 18:58:04.932458 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:04.932854 16547 external_mini_cluster.cc:1428] Setting key 9bcc7bff72c9a7f339184dcdf463216c
I20260812 18:58:04.934790 7181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: Bootstrap starting.
I20260812 18:58:04.937484 7181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:04.938349 7181 log.cc:824] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:04.940536 7181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: No bootstrap required, opened a new log
I20260812 18:58:04.942996 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:04.934586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.16.40.254:34851)
0812 18:58:04.935365 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:04.935384 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:04.935422 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:58:04.936202 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:04.936222 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:04.936724 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:04.937081 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:04.937565 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.938646 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.939567 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:04.940079 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:04.940259 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:04.941235 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:04.941240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:04.941253 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:04.941677 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:04.941735 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:04.941756 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:04.941870 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:04.941989 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 18:58:04.942268 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":158,"threads_started":1}
I20260812 18:58:04.945014 7181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } }
I20260812 18:58:04.945325 7181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:04.945431 7181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453261adcf684e729eb3ad5eddf7e46a, State: Initialized, Role: FOLLOWER
I20260812 18:58:04.945945 7181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } }
I20260812 18:58:04.946115 7181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:04.946213 7181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:04.946345 7181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:04.947563 7181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } }
I20260812 18:58:04.947952 7181 leader_election.cc:302] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453261adcf684e729eb3ad5eddf7e46a; no voters:
I20260812 18:58:04.948340 7181 leader_election.cc:288] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:04.948441 7187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:04.948796 7187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [term 1 LEADER]: Becoming Leader. State: Replica: 453261adcf684e729eb3ad5eddf7e46a, State: Running, Role: LEADER
I20260812 18:58:04.949149 7187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } }
I20260812 18:58:04.949875 7181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:04.951154 7189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453261adcf684e729eb3ad5eddf7e46a. Latest consensus state: current_term: 1 leader_uuid: "453261adcf684e729eb3ad5eddf7e46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } } }
I20260812 18:58:04.951323 7189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:04.951591 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453261adcf684e729eb3ad5eddf7e46a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453261adcf684e729eb3ad5eddf7e46a" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34851 } } }
I20260812 18:58:04.951697 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:04.952198 7198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:04.955487 7198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:04.961427 7198 catalog_manager.cc:1347] Generated new cluster ID: 54057c264edc46d9a335a3807d325e54
I20260812 18:58:04.961524 7198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:04.983889 7198 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:04.985381 7198 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:04.999010 7198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: Generated new TSK 0
I20260812 18:58:05.000039 7198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:05.010921 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.193:0
--local_ip_for_outbound_sockets=127.16.40.193
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--builtin_ntp_servers=127.16.40.212:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 18:58:05.142143 7210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:05.142441 7210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:05.142479 7210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:05.146692 7210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:05.146776 7210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:05.146836 7210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.193
I20260812 18:58:05.152194 7210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.40.193
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.193
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:05.153450 7210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:05.155033 7210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:05.166107 7217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:05.166112 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:05.166196 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:05.166992 7210 server_base.cc:1034] running on GCE node
I20260812 18:58:05.167583 7210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:05.168346 7210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:05.169588 7210 hybrid_clock.cc:648] HybridClock initialized: now 1786561085169565 us; error 38 us; skew 500 ppm
I20260812 18:58:05.171948 7210 webserver.cc:460] Webserver started at http://127.16.40.193:43197/ using document root <none> and password file <none>
I20260812 18:58:05.172726 7210 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:05.172847 7210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:05.173110 7210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:05.175446 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data/instance:
uuid: "6285a8a27aae4b9d8009c1773f2be729"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "2839c31e91592505e3864b74e6d2dd15"
server_key_iv: "4348a76d0429125040356f4a4fcc792b"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.176074 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance:
uuid: "6285a8a27aae4b9d8009c1773f2be729"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "2839c31e91592505e3864b74e6d2dd15"
server_key_iv: "4348a76d0429125040356f4a4fcc792b"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.180481 7210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:05.183465 7231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.185094 7210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:05.185313 7210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal
uuid: "6285a8a27aae4b9d8009c1773f2be729"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "2839c31e91592505e3864b74e6d2dd15"
server_key_iv: "4348a76d0429125040356f4a4fcc792b"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.185467 7210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:05.195802 7210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:05.196686 7210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:05.196903 7210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:05.197569 7210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:05.198634 7210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:05.198709 7210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.198783 7210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:05.198835 7210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.209154 7210 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.193:37803
I20260812 18:58:05.209235 7363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.193:37803 every 8 connection(s)
I20260812 18:58:05.210230 7210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/data/info.pb
I20260812 18:58:05.210561 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7210
I20260812 18:58:05.210708 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-0/wal/instance
I20260812 18:58:05.211082 16547 external_mini_cluster.cc:1428] Setting key 0213e934bb730f2fc9ac615eccf8f73f
I20260812 18:58:05.213457 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.194:0
--local_ip_for_outbound_sockets=127.16.40.194
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--builtin_ntp_servers=127.16.40.212:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:58:05.221556 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.212463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:59777 (local address 127.16.40.254:34851)
0812 18:58:05.212660 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.212665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.214231 (+ 1566us) server_negotiation.cc:407] Connection header received
0812 18:58:05.216146 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.216151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.216221 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.216337 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:05.217433 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.217989 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.219329 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.219571 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.219691 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.220984 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:05.220988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:05.220990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:05.221007 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:05.221036 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:05.221045 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:05.221125 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.221311 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.221386 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 18:58:05.222499 7368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.213065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:34851 (local address 127.16.40.193:59777)
0812 18:58:05.214062 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.214123 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.215885 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.216511 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.216518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.216684 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:05.217253 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.217268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.218148 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.218151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.219181 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.219192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.219747 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.219751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.219931 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.220686 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:05.220687 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:05.220733 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:05.220810 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:05.221141 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:05.221151 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:05.221158 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.221294 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.221618 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":178,"threads_started":1}
I20260812 18:58:05.224220 7364 heartbeater.cc:347] Connected to a master server at 127.16.40.254:34851
I20260812 18:58:05.224495 7364 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:05.225140 7364 heartbeater.cc:511] Master 127.16.40.254:34851 requested a full tablet report, sending...
I20260812 18:58:05.226933 7141 ts_manager.cc:194] Registered new tserver with Master: 6285a8a27aae4b9d8009c1773f2be729 (127.16.40.193:37803)
I20260812 18:58:05.228188 7141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.193:59777
W20260812 18:58:05.337121 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:05.337508 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:05.337589 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:05.342051 7371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:05.342182 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:05.342289 7371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.194
I20260812 18:58:05.347675 7371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.40.194
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.194
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:05.349092 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:05.350963 7371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:05.362712 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:05.362712 7385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:05.362861 7371 server_base.cc:1034] running on GCE node
W20260812 18:58:05.363111 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:05.363693 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:05.364490 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:05.365768 7371 hybrid_clock.cc:648] HybridClock initialized: now 1786561085365752 us; error 31 us; skew 500 ppm
I20260812 18:58:05.368274 7371 webserver.cc:460] Webserver started at http://127.16.40.194:38401/ using document root <none> and password file <none>
I20260812 18:58:05.369136 7371 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:05.369222 7371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:05.369490 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:05.372387 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data/instance:
uuid: "6d345d7eb9d04f78969e05ce521d93a8"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "3e286929378247eca19041a35d3b4adf"
server_key_iv: "15dbefcc405025acaa9c66770c4b9daf"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.373239 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance:
uuid: "6d345d7eb9d04f78969e05ce521d93a8"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "3e286929378247eca19041a35d3b4adf"
server_key_iv: "15dbefcc405025acaa9c66770c4b9daf"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.377835 7371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 18:58:05.380867 7391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.382479 7371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 18:58:05.382661 7371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal
uuid: "6d345d7eb9d04f78969e05ce521d93a8"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "3e286929378247eca19041a35d3b4adf"
server_key_iv: "15dbefcc405025acaa9c66770c4b9daf"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.382808 7371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:05.395906 7371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:05.396806 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:05.397037 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:05.397694 7371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:05.398772 7371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:05.398851 7371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.398936 7371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:05.398983 7371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.409543 7371 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.194:36759
I20260812 18:58:05.409652 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.194:36759 every 8 connection(s)
I20260812 18:58:05.411173 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/data/info.pb
I20260812 18:58:05.412918 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7371
I20260812 18:58:05.413062 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal/instance
I20260812 18:58:05.413450 16547 external_mini_cluster.cc:1428] Setting key 140243031da86dc68bba6b89771160f5
I20260812 18:58:05.416288 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.40.195:0
--local_ip_for_outbound_sockets=127.16.40.195
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--builtin_ntp_servers=127.16.40.212:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 18:58:05.425576 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.413974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:45559 (local address 127.16.40.254:34851)
0812 18:58:05.414135 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.414141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.417775 (+ 3634us) server_negotiation.cc:407] Connection header received
0812 18:58:05.419492 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.419496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.419558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.419650 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:05.421290 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.421967 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.423233 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.423480 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.423607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.424994 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:05.424998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:05.425001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:05.425026 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:05.425048 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:05.425057 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:05.425148 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.425335 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.425412 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 18:58:05.426594 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.414561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:34851 (local address 127.16.40.194:45559)
0812 18:58:05.417551 (+ 2990us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.417628 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.419234 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.420079 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.420089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.420247 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:05.421109 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.421133 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.422118 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.422122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.423082 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.423094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.423677 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.423684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.423876 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.424634 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:05.424637 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:05.424693 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:05.424776 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:05.425150 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:05.425163 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:05.425171 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.425321 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.425685 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"thread_start_us":212,"threads_started":1}
I20260812 18:58:05.428030 7521 heartbeater.cc:347] Connected to a master server at 127.16.40.254:34851
I20260812 18:58:05.428354 7521 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:05.429016 7521 heartbeater.cc:511] Master 127.16.40.254:34851 requested a full tablet report, sending...
I20260812 18:58:05.430241 7141 ts_manager.cc:194] Registered new tserver with Master: 6d345d7eb9d04f78969e05ce521d93a8 (127.16.40.194:36759)
I20260812 18:58:05.431033 7141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.194:45559
W20260812 18:58:05.544054 7525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:05.544387 7525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:05.544430 7525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:05.548893 7525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:05.548990 7525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:05.549052 7525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.40.195
I20260812 18:58:05.554541 7525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.40.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.40.195
--webserver_port=0
--tserver_master_addrs=127.16.40.254:34851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.40.195
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:05.555817 7525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:05.557320 7525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:05.568282 7533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:05.568284 7532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:05.568282 7535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:05.569041 7525 server_base.cc:1034] running on GCE node
I20260812 18:58:05.569566 7525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:05.570343 7525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:05.571528 7525 hybrid_clock.cc:648] HybridClock initialized: now 1786561085571504 us; error 37 us; skew 500 ppm
I20260812 18:58:05.573936 7525 webserver.cc:460] Webserver started at http://127.16.40.195:36685/ using document root <none> and password file <none>
I20260812 18:58:05.574591 7525 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:05.574659 7525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:05.574899 7525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:05.577229 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data/instance:
uuid: "904450649dfb48dab437daa4ddc83ec3"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "9b08183e36da7c5c4e21b52eb8d7588d"
server_key_iv: "11a672726721ada2b98c26087a928070"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.577821 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance:
uuid: "904450649dfb48dab437daa4ddc83ec3"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "9b08183e36da7c5c4e21b52eb8d7588d"
server_key_iv: "11a672726721ada2b98c26087a928070"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.582271 7525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:05.585294 7544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.586870 7525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:05.587064 7525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal
uuid: "904450649dfb48dab437daa4ddc83ec3"
format_stamp: "Formatted at 2026-08-12 18:58:05 on dist-test-slave-rnrw"
server_key: "9b08183e36da7c5c4e21b52eb8d7588d"
server_key_iv: "11a672726721ada2b98c26087a928070"
server_key_version: "encryptionkey@0"
I20260812 18:58:05.587211 7525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:05.618491 7525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:05.619302 7525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:05.619529 7525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:05.620246 7525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 18:58:05.621397 7525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 18:58:05.621472 7525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.621536 7525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 18:58:05.621590 7525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:05.631539 7525 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.195:39625
I20260812 18:58:05.631568 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.195:39625 every 8 connection(s)
I20260812 18:58:05.632602 7525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/data/info.pb
I20260812 18:58:05.635999 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7525
I20260812 18:58:05.636158 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal/instance
I20260812 18:58:05.636579 16547 external_mini_cluster.cc:1428] Setting key b12232141cf05676640b9f0492fd72a7
I20260812 18:58:05.643489 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.634737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.195:45493 (local address 127.16.40.254:34851)
0812 18:58:05.634883 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.634887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.636201 (+ 1314us) server_negotiation.cc:407] Connection header received
0812 18:58:05.638163 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.638166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.638247 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.638346 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:05.639591 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.640142 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.641495 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.641731 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.641827 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.643019 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:05.643022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:05.643024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:05.643038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:05.643060 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:05.643075 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:05.643137 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.643297 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.643359 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:58:05.644438 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.635101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:34851 (local address 127.16.40.195:45493)
0812 18:58:05.635958 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.636043 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.637883 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.638625 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.638633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.638730 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:05.639437 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.639464 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.640262 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.640264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.641352 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.641364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.641908 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.641911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.642075 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.642723 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 18:58:05.642724 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:05.642770 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 18:58:05.642856 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:05.643156 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:05.643163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:05.643168 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.643286 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.643551 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":139,"threads_started":1}
I20260812 18:58:05.645915 7680 heartbeater.cc:347] Connected to a master server at 127.16.40.254:34851
I20260812 18:58:05.646237 7680 heartbeater.cc:464] Registering TS with master...
I20260812 18:58:05.646833 7680 heartbeater.cc:511] Master 127.16.40.254:34851 requested a full tablet report, sending...
I20260812 18:58:05.648113 7141 ts_manager.cc:194] Registered new tserver with Master: 904450649dfb48dab437daa4ddc83ec3 (127.16.40.195:39625)
I20260812 18:58:05.648787 7141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.40.195:45493
I20260812 18:58:05.651319 16547 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 18:58:05.657963 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.653692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.16.40.254:34851)
0812 18:58:05.653868 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.653873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.654014 (+ 141us) server_negotiation.cc:407] Connection header received
0812 18:58:05.654169 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.654173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.654227 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.654314 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:05.654616 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.655155 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.656070 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.656299 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.656419 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.657407 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:05.657411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:05.657413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:05.657427 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:05.657449 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:05.657459 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:05.657555 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.657711 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.657784 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 18:58:05.661691 7141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 18:58:05.665210 7141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 18:58:05.681885 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.676035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:36759 (local address 127.0.0.1:55630)
0812 18:58:05.676671 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.676685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.676773 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.677798 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.677802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.677822 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.677910 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.677916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.679475 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.679479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.680460 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.680468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681377 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.681482 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681515 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.681558 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.681724 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":51,"thread_start_us":107,"threads_started":1}
I20260812 18:58:05.681890 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.675372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.16.40.193:37803)
0812 18:58:05.675969 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.675983 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.676158 (+ 175us) server_negotiation.cc:407] Connection header received
0812 18:58:05.676493 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.676513 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.676790 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.676930 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.677477 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.678680 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.680194 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681031 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681157 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681404 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.681576 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.681712 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":175,"threads_started":1}
I20260812 18:58:05.682427 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.675969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.16.40.195:39625)
0812 18:58:05.677200 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.677216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.677242 (+ 26us) server_negotiation.cc:407] Connection header received
0812 18:58:05.677316 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.677322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.677521 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.677679 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.677865 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.678610 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.680138 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681047 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681404 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.682169 (+ 765us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.682273 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":117,"threads_started":1}
I20260812 18:58:05.682394 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.676637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.16.40.194:36759)
0812 18:58:05.677297 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.677307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.677329 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:58:05.677393 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.677397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.677620 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.678062 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.678207 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.679325 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.680644 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681240 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681659 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681858 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.681908 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.682003 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":166,"threads_started":1}
I20260812 18:58:05.682714 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.675458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39625 (local address 127.0.0.1:34872)
0812 18:58:05.676206 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.676221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.676316 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.677657 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.677660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.677678 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.677768 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.677775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.678735 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.678739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.680002 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.680013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681681 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.681763 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681781 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.681854 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.681942 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":120,"threads_started":1}
I20260812 18:58:05.683301 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.675145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37803 (local address 127.0.0.1:35106)
0812 18:58:05.675993 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.676051 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.676284 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.677175 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.677181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.677208 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.677354 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.677361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.678823 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.678826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.680069 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.680080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.681177 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.681184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.681328 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.681390 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.681514 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.683134 (+ 1620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":1446,"thread_start_us":147,"threads_started":1}
I20260812 18:58:05.685788 7449 tablet_service.cc:1467] Processing CreateTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 (DEFAULT_TABLE table=test-table [id=4cbd8b93674349a0b13a6f0d7a93b2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:05.686313 7288 tablet_service.cc:1467] Processing CreateTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 (DEFAULT_TABLE table=test-table [id=4cbd8b93674349a0b13a6f0d7a93b2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:05.686313 7601 tablet_service.cc:1467] Processing CreateTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 (DEFAULT_TABLE table=test-table [id=4cbd8b93674349a0b13a6f0d7a93b2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 18:58:05.687031 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f939e3de93f74bbcbe0822b7f70b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:05.687377 7288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f939e3de93f74bbcbe0822b7f70b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:05.687377 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f939e3de93f74bbcbe0822b7f70b67e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:05.694898 7699 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Bootstrap starting.
I20260812 18:58:05.696903 7702 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Bootstrap starting.
I20260812 18:58:05.698277 7699 tablet_bootstrap.cc:654] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:05.698953 7702 tablet_bootstrap.cc:654] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:05.699189 7699 log.cc:824] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:05.699715 7702 log.cc:824] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:05.700662 7703 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Bootstrap starting.
I20260812 18:58:05.701659 7699 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: No bootstrap required, opened a new log
I20260812 18:58:05.701898 7699 ts_tablet_manager.cc:1397] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 18:58:05.701974 7702 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: No bootstrap required, opened a new log
I20260812 18:58:05.702155 7702 ts_tablet_manager.cc:1397] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 18:58:05.703195 7703 tablet_bootstrap.cc:654] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:05.703994 7703 log.cc:824] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:05.706180 7703 tablet_bootstrap.cc:492] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: No bootstrap required, opened a new log
I20260812 18:58:05.706436 7703 ts_tablet_manager.cc:1397] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 18:58:05.706741 7702 raft_consensus.cc:348] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.706741 7699 raft_consensus.cc:348] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.707062 7702 raft_consensus.cc:374] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:05.707062 7699 raft_consensus.cc:374] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:05.707124 7699 raft_consensus.cc:729] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d345d7eb9d04f78969e05ce521d93a8, State: Initialized, Role: FOLLOWER
I20260812 18:58:05.707130 7702 raft_consensus.cc:729] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6285a8a27aae4b9d8009c1773f2be729, State: Initialized, Role: FOLLOWER
I20260812 18:58:05.707676 7699 consensus_queue.cc:260] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.707676 7702 consensus_queue.cc:260] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.708921 7702 ts_tablet_manager.cc:1428] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:58:05.708921 7699 ts_tablet_manager.cc:1428] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:58:05.709389 7364 heartbeater.cc:503] Master 127.16.40.254:34851 was elected leader, sending a full tablet report...
I20260812 18:58:05.709740 7521 heartbeater.cc:503] Master 127.16.40.254:34851 was elected leader, sending a full tablet report...
I20260812 18:58:05.710973 7703 raft_consensus.cc:348] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.711309 7703 raft_consensus.cc:374] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:05.711383 7703 raft_consensus.cc:729] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904450649dfb48dab437daa4ddc83ec3, State: Initialized, Role: FOLLOWER
I20260812 18:58:05.711894 7703 consensus_queue.cc:260] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
W20260812 18:58:05.712312 7367 tablet.cc:2369] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:58:05.712771 7680 heartbeater.cc:503] Master 127.16.40.254:34851 was elected leader, sending a full tablet report...
I20260812 18:58:05.713070 7703 ts_tablet_manager.cc:1428] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 18:58:05.827361 7708 raft_consensus.cc:482] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:58:05.827658 7708 raft_consensus.cc:504] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.829332 7708 leader_election.cc:288] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6285a8a27aae4b9d8009c1773f2be729 (127.16.40.193:37803), 904450649dfb48dab437daa4ddc83ec3 (127.16.40.195:39625)
I20260812 18:58:05.833757 7710 raft_consensus.cc:482] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 18:58:05.833954 7710 raft_consensus.cc:504] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.834575 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.829584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37803 (local address 127.16.40.194:39633)
0812 18:58:05.829927 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.829943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.830049 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.830367 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.830372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.830410 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.830585 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.830598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.831769 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.831773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.833129 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.833145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.834049 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.834053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.834223 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.834252 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.834341 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.834426 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":26,"spinlock_wait_cycles":1792}
I20260812 18:58:05.834610 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.829896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:39633 (local address 127.16.40.193:37803)
0812 18:58:05.830063 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.830070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.830100 (+ 30us) server_negotiation.cc:407] Connection header received
0812 18:58:05.830175 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.830178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.830236 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.830337 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.830764 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.831654 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.833291 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.833902 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.834026 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.834078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.834380 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.834480 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:58:05.834666 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.830037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.194:35425 (local address 127.16.40.195:39625)
0812 18:58:05.830203 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.830210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.830339 (+ 129us) server_negotiation.cc:407] Connection header received
0812 18:58:05.830541 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.830546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.830635 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.830713 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.831055 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.831669 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.833153 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.833805 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.833934 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.834022 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.834472 (+ 450us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.834554 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 18:58:05.835183 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.829653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39625 (local address 127.16.40.194:35425)
0812 18:58:05.830245 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.830272 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.830426 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.830824 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.830827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.830842 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.830940 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.830944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.831839 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.831845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.833017 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.833028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.834160 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.834163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.834284 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.834303 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.834957 (+ 654us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.835042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":22,"spinlock_wait_cycles":8192,"thread_start_us":115,"threads_started":1}
I20260812 18:58:05.835353 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6d345d7eb9d04f78969e05ce521d93a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285a8a27aae4b9d8009c1773f2be729" is_pre_election: true
I20260812 18:58:05.835449 7710 leader_election.cc:288] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d345d7eb9d04f78969e05ce521d93a8 (127.16.40.194:36759), 904450649dfb48dab437daa4ddc83ec3 (127.16.40.195:39625)
I20260812 18:58:05.835584 7316 raft_consensus.cc:2454] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d345d7eb9d04f78969e05ce521d93a8 in term 0.
I20260812 18:58:05.835891 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6d345d7eb9d04f78969e05ce521d93a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904450649dfb48dab437daa4ddc83ec3" is_pre_election: true
I20260812 18:58:05.836232 7625 raft_consensus.cc:2454] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d345d7eb9d04f78969e05ce521d93a8 in term 0.
I20260812 18:58:05.836905 7394 leader_election.cc:302] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d345d7eb9d04f78969e05ce521d93a8, 904450649dfb48dab437daa4ddc83ec3; no voters:
I20260812 18:58:05.838164 7708 raft_consensus.cc:2781] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 18:58:05.838269 7708 raft_consensus.cc:482] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 18:58:05.838330 7708 raft_consensus.cc:3037] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:05.839957 7708 raft_consensus.cc:504] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.840786 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6d345d7eb9d04f78969e05ce521d93a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904450649dfb48dab437daa4ddc83ec3"
I20260812 18:58:05.840992 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.836167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:45705 (local address 127.16.40.195:39625)
0812 18:58:05.836272 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.836276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.836709 (+ 433us) server_negotiation.cc:407] Connection header received
0812 18:58:05.836806 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.836809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.836850 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.836957 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.837301 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.837906 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.839650 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.840215 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.840303 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.840375 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.840786 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.840866 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 18:58:05.840938 7625 raft_consensus.cc:3037] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:05.841059 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.836004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.195:39625 (local address 127.16.40.193:45705)
0812 18:58:05.836606 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.836628 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.836729 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.836969 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.836972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.836989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.837121 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.837128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.838037 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.838040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.839509 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.839521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.840482 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.840487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.840626 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.840652 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.840763 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.840905 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":338,"thread_start_us":110,"threads_started":1}
I20260812 18:58:05.841223 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.835802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.194:36759 (local address 127.16.40.193:34497)
0812 18:58:05.836132 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.836149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:05.836245 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:05.837602 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:05.837605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:05.837625 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 18:58:05.837730 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.837736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.838713 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.838717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.839773 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:05.839782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.840720 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.840724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:05.840875 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.840904 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 18:58:05.840986 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 18:58:05.841070 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260812 18:58:05.841717 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.836675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.40.193:34497 (local address 127.16.40.194:36759)
0812 18:58:05.837342 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.837348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.837364 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:58:05.837415 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.837420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.837480 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.837573 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 18:58:05.837858 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.838563 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.839907 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.840445 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.840590 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.840637 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.841498 (+ 861us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.841593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577}
I20260812 18:58:05.842350 7625 raft_consensus.cc:2454] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d345d7eb9d04f78969e05ce521d93a8 in term 1.
I20260812 18:58:05.842763 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6285a8a27aae4b9d8009c1773f2be729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904450649dfb48dab437daa4ddc83ec3" is_pre_election: true
I20260812 18:58:05.842790 7316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6d345d7eb9d04f78969e05ce521d93a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6285a8a27aae4b9d8009c1773f2be729"
I20260812 18:58:05.842968 7625 raft_consensus.cc:2379] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6285a8a27aae4b9d8009c1773f2be729 in current term 1: Already voted for candidate 6d345d7eb9d04f78969e05ce521d93a8 in this term.
I20260812 18:58:05.842954 7316 raft_consensus.cc:3037] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:05.844149 7316 raft_consensus.cc:2454] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d345d7eb9d04f78969e05ce521d93a8 in term 1.
I20260812 18:58:05.844599 7396 leader_election.cc:302] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6285a8a27aae4b9d8009c1773f2be729, 6d345d7eb9d04f78969e05ce521d93a8, 904450649dfb48dab437daa4ddc83ec3; no voters:
I20260812 18:58:05.844658 7708 leader_election.cc:288] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6285a8a27aae4b9d8009c1773f2be729 (127.16.40.193:37803), 904450649dfb48dab437daa4ddc83ec3 (127.16.40.195:39625)
I20260812 18:58:05.844806 7708 raft_consensus.cc:2781] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:05.844892 7708 raft_consensus.cc:686] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d345d7eb9d04f78969e05ce521d93a8, State: Running, Role: LEADER
I20260812 18:58:05.845141 7471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f939e3de93f74bbcbe0822b7f70b67e4" candidate_uuid: "6285a8a27aae4b9d8009c1773f2be729" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d345d7eb9d04f78969e05ce521d93a8" is_pre_election: true
I20260812 18:58:05.845248 7708 consensus_queue.cc:237] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } }
I20260812 18:58:05.845860 7234 leader_election.cc:302] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6285a8a27aae4b9d8009c1773f2be729; no voters: 6d345d7eb9d04f78969e05ce521d93a8, 904450649dfb48dab437daa4ddc83ec3
I20260812 18:58:05.846187 7710 raft_consensus.cc:2726] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 18:58:05.849020 7141 catalog_manager.cc:5495] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d345d7eb9d04f78969e05ce521d93a8 (127.16.40.194). New cstate: current_term: 1 leader_uuid: "6d345d7eb9d04f78969e05ce521d93a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d345d7eb9d04f78969e05ce521d93a8" member_type: VOTER last_known_addr { host: "127.16.40.194" port: 36759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 } health_report { overall_health: UNKNOWN } } }
W20260812 18:58:05.883774 7681 tablet.cc:2369] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:58:05.895318 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.891465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.16.40.194:36759)
0812 18:58:05.891633 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:05.891639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:05.891661 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:58:05.891782 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:05.891784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:05.891844 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:05.891938 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:05.892245 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.892861 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.893816 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:05.894053 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:05.894151 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:05.894470 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:05.894951 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:05.895057 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:05.895139 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 18:58:05.895194 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:58:05.901887 7625 raft_consensus.cc:1261] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 1 FOLLOWER]: Refusing update from remote peer 6d345d7eb9d04f78969e05ce521d93a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:05.901940 7316 raft_consensus.cc:1261] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 1 FOLLOWER]: Refusing update from remote peer 6d345d7eb9d04f78969e05ce521d93a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:05.902666 7715 consensus_queue.cc:1035] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6285a8a27aae4b9d8009c1773f2be729" member_type: VOTER last_known_addr { host: "127.16.40.193" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:05.902895 7708 consensus_queue.cc:1035] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "904450649dfb48dab437daa4ddc83ec3" member_type: VOTER last_known_addr { host: "127.16.40.195" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:05.911118 7720 mvcc.cc:205] Tried to move back new op lower bound from 7317754207845892096 to 7317754207625248768. Current Snapshot: MvccSnapshot[applied={T|T < 7317754207845892096}]
I20260812 18:58:05.921694 7141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:05.921929 7141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 18:58:05.925097 7141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 453261adcf684e729eb3ad5eddf7e46a: Sending DeleteTablet for 3 replicas of tablet f939e3de93f74bbcbe0822b7f70b67e4
I20260812 18:58:05.926246 7288 tablet_service.cc:1514] Processing DeleteTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:05 UTC) from {username='slave'} at 127.0.0.1:35106
I20260812 18:58:05.926340 7449 tablet_service.cc:1514] Processing DeleteTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:05 UTC) from {username='slave'} at 127.0.0.1:55630
I20260812 18:58:05.926327 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet f939e3de93f74bbcbe0822b7f70b67e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 18:58:05 UTC) from {username='slave'} at 127.0.0.1:34872
I20260812 18:58:05.926888 7731 tablet_replica.cc:330] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729: stopping tablet replica
I20260812 18:58:05.927268 7731 raft_consensus.cc:2229] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:05.927692 7731 raft_consensus.cc:2258] T f939e3de93f74bbcbe0822b7f70b67e4 P 6285a8a27aae4b9d8009c1773f2be729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:05.928408 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7210
I20260812 18:58:05.931591 7733 tablet_replica.cc:330] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: stopping tablet replica
I20260812 18:58:05.931972 7733 raft_consensus.cc:2229] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 18:58:05.932406 7733 raft_consensus.cc:2258] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:05.932809 7734 tablet_replica.cc:330] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: stopping tablet replica
I20260812 18:58:05.933117 7734 raft_consensus.cc:2229] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 18:58:05.933465 7734 raft_consensus.cc:2258] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 18:58:05.941555 7733 ts_tablet_manager.cc:1905] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:05.941555 7734 ts_tablet_manager.cc:1905] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 18:58:05.944653 7734 ts_tablet_manager.cc:1918] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:05.944653 7733 ts_tablet_manager.cc:1918] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 18:58:05.944788 7734 log.cc:1196] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-2/wal/wals/f939e3de93f74bbcbe0822b7f70b67e4
I20260812 18:58:05.944968 7733 log.cc:1196] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Deleting WAL directory at /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786560811542710-16547-0/minicluster-data/ts-1/wal/wals/f939e3de93f74bbcbe0822b7f70b67e4
I20260812 18:58:05.945142 7734 ts_tablet_manager.cc:1939] T f939e3de93f74bbcbe0822b7f70b67e4 P 904450649dfb48dab437daa4ddc83ec3: Deleting consensus metadata
I20260812 18:58:05.945500 7733 ts_tablet_manager.cc:1939] T f939e3de93f74bbcbe0822b7f70b67e4 P 6d345d7eb9d04f78969e05ce521d93a8: Deleting consensus metadata
W20260812 18:58:05.945611 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.40.193:37803 (error 108)
I20260812 18:58:05.947124 7127 catalog_manager.cc:4841] TS 6d345d7eb9d04f78969e05ce521d93a8 (127.16.40.194:36759): tablet f939e3de93f74bbcbe0822b7f70b67e4 (table test-table [id=4cbd8b93674349a0b13a6f0d7a93b2d5]) successfully deleted
I20260812 18:58:05.947773 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7371
I20260812 18:58:05.947784 7126 catalog_manager.cc:4841] TS 904450649dfb48dab437daa4ddc83ec3 (127.16.40.195:39625): tablet f939e3de93f74bbcbe0822b7f70b67e4 (table test-table [id=4cbd8b93674349a0b13a6f0d7a93b2d5]) successfully deleted
I20260812 18:58:05.954802 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:05.954334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.193:37803 (local address 127.0.0.1:35118)
0812 18:58:05.954590 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:05.954686 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.193:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260812 18:58:05.955056 7127 catalog_manager.cc:4584] TS 6285a8a27aae4b9d8009c1773f2be729 (127.16.40.193:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f939e3de93f74bbcbe0822b7f70b67e4: Network error: Client connection negotiation failed: client connection to 127.16.40.193:37803: connect: Connection refused (error 111)
I20260812 18:58:05.961493 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7525
I20260812 18:58:05.971917 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7103
2026-08-12T18:58:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1283 ms)
[----------] 6 tests from EncryptionPolicyTest (11698 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:06 dist-test-slave-rnrw krb5kdc[7738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:06 dist-test-slave-rnrw krb5kdc[7738](info): set up 2 sockets
Aug 12 18:58:06 dist-test-slave-rnrw krb5kdc[7738](info): commencing operation
krb5kdc: starting...
W20260812 18:58:07.138496 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:07.280018 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:07.404187 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:07.404716 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:07.404852 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:07.409214 7751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:07.409314 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:07.409353 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:07.409422 7751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:07.409479 7751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:07.415249 7751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:07.416599 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:07.417740 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:07.427153 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:07.427138 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:07.427191 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:07.427582 7751 server_base.cc:1034] running on GCE node
I20260812 18:58:07.428470 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:07.429817 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:07.431015 7751 hybrid_clock.cc:648] HybridClock initialized: now 1786561087430992 us; error 36 us; skew 500 ppm
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:07.438129 7751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:07.439378 7751 webserver.cc:460] Webserver started at http://127.0.0.1:43847/ using document root <none> and password file <none>
I20260812 18:58:07.440165 7751 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:07.440240 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:07.440521 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:07.443097 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "2f9a39dcd4924bc599dfe26692ae68b4"
format_stamp: "Formatted at 2026-08-12 18:58:07 on dist-test-slave-rnrw"
server_key: "be46af94781079c219ebfdd89a6feccb"
server_key_iv: "d69dcf9990d55e50b3cafd9797ebe77b"
server_key_version: "encryptionkey@0"
I20260812 18:58:07.443799 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "2f9a39dcd4924bc599dfe26692ae68b4"
format_stamp: "Formatted at 2026-08-12 18:58:07 on dist-test-slave-rnrw"
server_key: "be46af94781079c219ebfdd89a6feccb"
server_key_iv: "d69dcf9990d55e50b3cafd9797ebe77b"
server_key_version: "encryptionkey@0"
I20260812 18:58:07.448529 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:58:07.451573 7767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:07.453361 7751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:07.453514 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "2f9a39dcd4924bc599dfe26692ae68b4"
format_stamp: "Formatted at 2026-08-12 18:58:07 on dist-test-slave-rnrw"
server_key: "be46af94781079c219ebfdd89a6feccb"
server_key_iv: "d69dcf9990d55e50b3cafd9797ebe77b"
server_key_version: "encryptionkey@0"
I20260812 18:58:07.453685 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:07.473482 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:07.477427 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:07.477713 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:07.485690 7751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39035
I20260812 18:58:07.485885 7829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39035 every 8 connection(s)
I20260812 18:58:07.486871 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:07.488893 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7751
I20260812 18:58:07.489015 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:07.489332 16547 external_mini_cluster.cc:1428] Setting key 946c85be523a53e833c1d7f2b045c6e1
I20260812 18:58:07.490799 7830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:07.498549 7830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4: Bootstrap starting.
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 18:58:07.501577 7830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:07.502440 7830 log.cc:824] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:07.505244 7830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4: No bootstrap required, opened a new log
I20260812 18:58:07.507254 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:07.491101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51645 (local address 127.0.0.1:39035)
0812 18:58:07.491599 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:07.491623 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:07.491671 (+ 48us) server_negotiation.cc:407] Connection header received
0812 18:58:07.492607 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:07.492634 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:07.493151 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:07.493566 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:07.494738 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:07.496266 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:07.497377 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:07.498126 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:07.498305 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:07.501428 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:07.501435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:07.501454 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:07.501492 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:07.503511 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:07.504240 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:07.504246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:07.504253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:07.504386 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:07.504822 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:07.504825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:07.504826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:07.505296 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:07.505573 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:07.505978 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 18:58:07.506310 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260812 18:58:07.510341 7830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } }
I20260812 18:58:07.510738 7830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:07.510846 7830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9a39dcd4924bc599dfe26692ae68b4, State: Initialized, Role: FOLLOWER
I20260812 18:58:07.511533 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:07.511478 7830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } }
I20260812 18:58:07.511682 7830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:07.511765 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:07.511885 7830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:07.513365 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } }
I20260812 18:58:07.513827 7830 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9a39dcd4924bc599dfe26692ae68b4; no voters:
I20260812 18:58:07.514281 7830 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:07.514390 7843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:07.514650 7843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f9a39dcd4924bc599dfe26692ae68b4, State: Running, Role: LEADER
I20260812 18:58:07.515034 7843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } }
I20260812 18:58:07.515906 7830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:07.516584 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9a39dcd4924bc599dfe26692ae68b4. Latest consensus state: current_term: 1 leader_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } } }
I20260812 18:58:07.516721 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:07.517104 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a39dcd4924bc599dfe26692ae68b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39035 } } }
I20260812 18:58:07.517225 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:07.518237 7848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:07.522611 7848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:07.524158 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:07.513975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34279 (local address 127.0.0.1:39035)
0812 18:58:07.514183 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:07.514192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:07.514613 (+ 421us) server_negotiation.cc:407] Connection header received
0812 18:58:07.514668 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:07.514672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:07.514730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:07.514803 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:07.515857 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:07.516673 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:07.517826 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:07.518189 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:07.518778 (+ 589us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:07.520158 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:07.520162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:07.520164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:07.520189 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:07.521935 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:07.522567 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:07.522571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:07.522572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:07.522658 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:07.523154 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:07.523158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:07.523160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:07.523383 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:07.523564 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:07.523872 (+ 308us) server_negotiation.cc:299] Negotiation successful
0812 18:58:07.523983 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 18:58:07.530046 7848 catalog_manager.cc:1347] Generated new cluster ID: c21662869c1544cd82781e9371a8a720
I20260812 18:58:07.530157 7848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:07.540203 7848 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:07.541373 7848 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:07.560179 7848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9a39dcd4924bc599dfe26692ae68b4: Generated new TSK 0
I20260812 18:58:07.561187 7848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:07.568264 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7751
W20260812 18:58:07.578938 7736 connection.cc:535] client connection to 127.0.0.1:39035 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39035): Connection reset by peer (error 104)
2026-08-12T18:58:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1595 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7869](info): set up 2 sockets
Aug 12 18:58:07 dist-test-slave-rnrw krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260812 18:58:08.706426 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:08 dist-test-slave-rnrw krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:08.853047 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:08.974094 7884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:08.974475 7884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:08.974566 7884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:08.978933 7884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:08.979063 7884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:08.979103 7884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:08.979142 7884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:08.979177 7884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:08.985036 7884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:08.986313 7884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:08.987443 7884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:08.997234 7889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:08.997459 7884 server_base.cc:1034] running on GCE node
W20260812 18:58:08.997246 7892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:08.997288 7890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:08.998286 7884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:08.999603 7884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:09.000809 7884 hybrid_clock.cc:648] HybridClock initialized: now 1786561089000774 us; error 54 us; skew 500 ppm
Aug 12 18:58:08 dist-test-slave-rnrw krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:09.004043 7884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:09.005160 7884 webserver.cc:460] Webserver started at http://127.0.0.1:36759/ using document root <none> and password file <none>
I20260812 18:58:09.005776 7884 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:09.005844 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:09.006034 7884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:09.008343 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "3d6e69a1351d4bed903fb384f7f454c6"
format_stamp: "Formatted at 2026-08-12 18:58:09 on dist-test-slave-rnrw"
server_key: "1498b8b4a2d38394c64cf06cbb593136"
server_key_iv: "fdca309830589911ab762ff5bde7e42d"
server_key_version: "encryptionkey@0"
I20260812 18:58:09.009037 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "3d6e69a1351d4bed903fb384f7f454c6"
format_stamp: "Formatted at 2026-08-12 18:58:09 on dist-test-slave-rnrw"
server_key: "1498b8b4a2d38394c64cf06cbb593136"
server_key_iv: "fdca309830589911ab762ff5bde7e42d"
server_key_version: "encryptionkey@0"
I20260812 18:58:09.013489 7884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:09.016474 7900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:09.018088 7884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:09.018322 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "3d6e69a1351d4bed903fb384f7f454c6"
format_stamp: "Formatted at 2026-08-12 18:58:09 on dist-test-slave-rnrw"
server_key: "1498b8b4a2d38394c64cf06cbb593136"
server_key_iv: "fdca309830589911ab762ff5bde7e42d"
server_key_version: "encryptionkey@0"
I20260812 18:58:09.018481 7884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:09.031894 7884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:09.035848 7884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:09.036100 7884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:09.044677 7884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32937
I20260812 18:58:09.044713 7961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32937 every 8 connection(s)
I20260812 18:58:09.045869 7884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:09.049412 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:09.051657 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 7884
I20260812 18:58:09.051792 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:09.052145 16547 external_mini_cluster.cc:1428] Setting key 3eb2929e88f9a9beec66da4691731b1c
I20260812 18:58:09.057776 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6: Bootstrap starting.
I20260812 18:58:09.060647 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:09.061585 7963 log.cc:824] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:09.062980 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:09.053916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42713 (local address 127.0.0.1:32937)
0812 18:58:09.054588 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:09.054608 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:09.054658 (+ 50us) server_negotiation.cc:407] Connection header received
0812 18:58:09.055780 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:09.055806 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:09.056252 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:09.056702 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:09.057161 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:09.058212 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:09.059171 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:09.059684 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:09.059877 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:09.060924 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:09.060929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:09.060945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:09.061373 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:09.061445 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:09.061469 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:09.061629 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:09.061814 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 18:58:09.061978 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":191,"threads_started":1}
I20260812 18:58:09.064312 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6: No bootstrap required, opened a new log
I20260812 18:58:09.066782 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:09.069525 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260812 18:58:09.069861 7963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:09.069940 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6e69a1351d4bed903fb384f7f454c6, State: Initialized, Role: FOLLOWER
I20260812 18:58:09.070492 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260812 18:58:09.070706 7963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:09.070794 7963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:09.070911 7963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:09.072275 7963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260812 18:58:09.072816 7963 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6e69a1351d4bed903fb384f7f454c6; no voters:
I20260812 18:58:09.073331 7963 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:09.073489 7975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:09.073794 7975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6e69a1351d4bed903fb384f7f454c6, State: Running, Role: LEADER
I20260812 18:58:09.074188 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } }
I20260812 18:58:09.074736 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:09.069216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44215 (local address 127.0.0.1:32937)
0812 18:58:09.069347 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:09.069350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:09.069655 (+ 305us) server_negotiation.cc:407] Connection header received
0812 18:58:09.069829 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:09.069833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:09.069910 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:09.069984 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:09.070498 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:09.071364 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:09.072288 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:09.072733 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:09.072871 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:09.074079 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:09.074084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:09.074087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:09.074107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:09.074144 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:09.074161 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:09.074266 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:09.074533 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 18:58:09.074582 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 18:58:09.075182 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:09.075635 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6e69a1351d4bed903fb384f7f454c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } } }
I20260812 18:58:09.075773 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:09.076020 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6e69a1351d4bed903fb384f7f454c6. Latest consensus state: current_term: 1 leader_uuid: "3d6e69a1351d4bed903fb384f7f454c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6e69a1351d4bed903fb384f7f454c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32937 } } }
I20260812 18:58:09.076115 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:09.076735 7980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:09.080348 7980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:09.087600 7980 catalog_manager.cc:1347] Generated new cluster ID: 1bd64d3490fe4b6e80a0a1457f117f1f
I20260812 18:58:09.087710 7980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:09.094997 7980 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:09.096154 7980 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:09.108770 7980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6e69a1351d4bed903fb384f7f454c6: Generated new TSK 0
I20260812 18:58:09.109701 7980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:09.120517 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 7884
W20260812 18:58:09.130457 7867 connection.cc:535] client connection to 127.0.0.1:32937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32937): Connection reset by peer (error 104)
2026-08-12T18:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:09 dist-test-slave-rnrw krb5kdc[8002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:09 dist-test-slave-rnrw krb5kdc[8002](info): set up 2 sockets
Aug 12 18:58:09 dist-test-slave-rnrw krb5kdc[8002](info): commencing operation
krb5kdc: starting...
W20260812 18:58:10.264485 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:10 dist-test-slave-rnrw krb5kdc[8002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:10.412317 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:10.541923 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:10.542305 8016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:10.542393 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:10.546650 8016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:10.546774 8016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:10.546816 8016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:10.546855 8016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:10.546893 8016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:10.552934 8016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:10.554286 8016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:10.555461 8016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:10.564925 8027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:10.564968 8021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:10.564925 8024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:10.565383 8016 server_base.cc:1034] running on GCE node
I20260812 18:58:10.566061 8016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:10.567348 8016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:10.568622 8016 hybrid_clock.cc:648] HybridClock initialized: now 1786561090568601 us; error 38 us; skew 500 ppm
Aug 12 18:58:10 dist-test-slave-rnrw krb5kdc[8002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:10.571674 8016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:10.572741 8016 webserver.cc:460] Webserver started at http://127.0.0.1:45205/ using document root <none> and password file <none>
I20260812 18:58:10.573335 8016 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:10.573405 8016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:10.573596 8016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:10.575744 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "72229b2fe4df454f97fc79e6647d9a66"
format_stamp: "Formatted at 2026-08-12 18:58:10 on dist-test-slave-rnrw"
server_key: "23ae0df8782c2b48d1a4de85ec0b153a"
server_key_iv: "fe429b706c919c044bf98130a1786dde"
server_key_version: "encryptionkey@0"
I20260812 18:58:10.576318 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "72229b2fe4df454f97fc79e6647d9a66"
format_stamp: "Formatted at 2026-08-12 18:58:10 on dist-test-slave-rnrw"
server_key: "23ae0df8782c2b48d1a4de85ec0b153a"
server_key_iv: "fe429b706c919c044bf98130a1786dde"
server_key_version: "encryptionkey@0"
I20260812 18:58:10.580657 8016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:10.583582 8035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:10.585312 8016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:10.585502 8016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "72229b2fe4df454f97fc79e6647d9a66"
format_stamp: "Formatted at 2026-08-12 18:58:10 on dist-test-slave-rnrw"
server_key: "23ae0df8782c2b48d1a4de85ec0b153a"
server_key_iv: "fe429b706c919c044bf98130a1786dde"
server_key_version: "encryptionkey@0"
I20260812 18:58:10.585628 8016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:10.591081 8016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:10.594777 8016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:10.595024 8016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:10.602926 8016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37285
I20260812 18:58:10.602944 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37285 every 8 connection(s)
I20260812 18:58:10.604022 8016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:10.607316 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:10.610823 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8016
I20260812 18:58:10.611001 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:10.611307 16547 external_mini_cluster.cc:1428] Setting key 098427d252060162fb8ef4afc6213f10
I20260812 18:58:10.614310 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: Bootstrap starting.
I20260812 18:58:10.617399 8099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:10.617373 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:10.612958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39319 (local address 127.0.0.1:37285)
0812 18:58:10.613489 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:10.613506 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:10.613552 (+ 46us) server_negotiation.cc:407] Connection header received
0812 18:58:10.614225 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:10.614243 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:10.614627 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:10.614947 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:10.615938 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:10.615943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:10.615958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:10.615970 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:10.616039 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:10.616060 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:10.616186 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:10.616314 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 18:58:10.616585 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":183,"threads_started":1}
I20260812 18:58:10.618295 8099 log.cc:824] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:10.620821 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:10.621014 8099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: No bootstrap required, opened a new log
I20260812 18:58:10.625527 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:10.623260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42587 (local address 127.0.0.1:37285)
0812 18:58:10.623471 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:10.623475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:10.623746 (+ 271us) server_negotiation.cc:407] Connection header received
0812 18:58:10.623941 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:10.623945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:10.623992 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:10.624094 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:10.625017 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:10.625021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:10.625025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:10.625045 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:10.625082 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:10.625095 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:10.625209 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:10.625332 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 18:58:10.625381 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:58:10.625311 8099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 18:58:10.625703 8099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:10.625787 8099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72229b2fe4df454f97fc79e6647d9a66, State: Initialized, Role: FOLLOWER
I20260812 18:58:10.626317 8099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 18:58:10.626458 8099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:10.626559 8099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:10.626675 8099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:10.627701 8099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 18:58:10.628041 8099 leader_election.cc:302] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72229b2fe4df454f97fc79e6647d9a66; no voters:
I20260812 18:58:10.628340 8099 leader_election.cc:288] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:10.628526 8110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:10.628818 8110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [term 1 LEADER]: Becoming Leader. State: Replica: 72229b2fe4df454f97fc79e6647d9a66, State: Running, Role: LEADER
I20260812 18:58:10.629194 8110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260812 18:58:10.629818 8099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:10.631058 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72229b2fe4df454f97fc79e6647d9a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260812 18:58:10.631213 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:10.631587 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72229b2fe4df454f97fc79e6647d9a66. Latest consensus state: current_term: 1 leader_uuid: "72229b2fe4df454f97fc79e6647d9a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72229b2fe4df454f97fc79e6647d9a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260812 18:58:10.631784 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66 [sys.catalog]: This master's current role is: LEADER
W20260812 18:58:10.635263 8129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:10.635370 8129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:10.635473 8123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:10.636267 8123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:10.642494 8123 catalog_manager.cc:1347] Generated new cluster ID: 5c177a2aa7684805b6b627fab7135603
I20260812 18:58:10.642594 8123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:10.654198 8123 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:10.654623 8123 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:10.661339 8123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72229b2fe4df454f97fc79e6647d9a66: Generated new TSK 0
I20260812 18:58:10.662173 8123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:10.671119 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8016
2026-08-12T18:58:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:10 dist-test-slave-rnrw krb5kdc[8135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:10 dist-test-slave-rnrw krb5kdc[8135](info): set up 2 sockets
Aug 12 18:58:10 dist-test-slave-rnrw krb5kdc[8135](info): commencing operation
krb5kdc: starting...
W20260812 18:58:11.859191 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:11 dist-test-slave-rnrw krb5kdc[8135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:12.000183 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:12.120970 8155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:12.121341 8155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:12.121434 8155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:12.125756 8155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:12.125882 8155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:12.125921 8155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:12.125962 8155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:12.125999 8155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:12.131793 8155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:12.133133 8155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:12.134258 8155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:12.143390 8160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:12.143404 8163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:12.143390 8161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:12.143882 8155 server_base.cc:1034] running on GCE node
I20260812 18:58:12.144604 8155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:12.145902 8155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:12.147081 8155 hybrid_clock.cc:648] HybridClock initialized: now 1786561092147067 us; error 28 us; skew 500 ppm
Aug 12 18:58:12 dist-test-slave-rnrw krb5kdc[8135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:12.150319 8155 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:12.151379 8155 webserver.cc:460] Webserver started at http://127.0.0.1:36145/ using document root <none> and password file <none>
I20260812 18:58:12.152022 8155 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:12.152089 8155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:12.152340 8155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:12.154601 8155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "9a88f89dc9d544849c18172bf80b858e"
format_stamp: "Formatted at 2026-08-12 18:58:12 on dist-test-slave-rnrw"
server_key: "3ae31dd3a9283cd2cb989ef6582c4e3a"
server_key_iv: "47471e9a0b6841ece0dc26600b5afcc3"
server_key_version: "encryptionkey@0"
I20260812 18:58:12.155226 8155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "9a88f89dc9d544849c18172bf80b858e"
format_stamp: "Formatted at 2026-08-12 18:58:12 on dist-test-slave-rnrw"
server_key: "3ae31dd3a9283cd2cb989ef6582c4e3a"
server_key_iv: "47471e9a0b6841ece0dc26600b5afcc3"
server_key_version: "encryptionkey@0"
I20260812 18:58:12.159372 8155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:58:12.162356 8174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:12.163897 8155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:58:12.164072 8155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "9a88f89dc9d544849c18172bf80b858e"
format_stamp: "Formatted at 2026-08-12 18:58:12 on dist-test-slave-rnrw"
server_key: "3ae31dd3a9283cd2cb989ef6582c4e3a"
server_key_iv: "47471e9a0b6841ece0dc26600b5afcc3"
server_key_version: "encryptionkey@0"
I20260812 18:58:12.164216 8155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:12.177075 8155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:12.180907 8155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:12.181142 8155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:12.189361 8155 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37073
I20260812 18:58:12.189392 8240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37073 every 8 connection(s)
I20260812 18:58:12.190476 8155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:12.193715 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:12.198695 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8155
I20260812 18:58:12.198860 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:12.199162 16547 external_mini_cluster.cc:1428] Setting key 10c937f9830216f8e1b2b4dc72066410
I20260812 18:58:12.201052 8241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e: Bootstrap starting.
I20260812 18:58:12.203715 8241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:12.204704 8241 log.cc:824] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:12.207324 8241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e: No bootstrap required, opened a new log
Aug 12 18:58:12 dist-test-slave-rnrw krb5kdc[8135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 18:58:12.212267 8241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260812 18:58:12.212743 8241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:12.212812 8241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a88f89dc9d544849c18172bf80b858e, State: Initialized, Role: FOLLOWER
I20260812 18:58:12.213423 8241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260812 18:58:12.213666 8241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:12.213778 8241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:12.213899 8241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:12.215371 8241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260812 18:58:12.215862 8241 leader_election.cc:302] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a88f89dc9d544849c18172bf80b858e; no voters:
I20260812 18:58:12.216353 8241 leader_election.cc:288] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:12.216495 8246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:12.216804 8246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [term 1 LEADER]: Becoming Leader. State: Replica: 9a88f89dc9d544849c18172bf80b858e, State: Running, Role: LEADER
I20260812 18:58:12.217242 8246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260812 18:58:12.217567 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:12.200928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36175 (local address 127.0.0.1:37073)
0812 18:58:12.201518 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:12.201536 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:12.201578 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:58:12.202361 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:12.202382 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:12.202810 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:12.203222 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:12.204217 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:12.205550 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:12.206725 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:12.207480 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:12.207669 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:12.210882 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:12.210890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:12.210917 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:12.210966 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:12.213378 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:12.214141 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:12.214151 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:12.214161 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:12.214347 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:12.214741 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:12.214745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:12.214747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:12.215434 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:12.215674 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:12.216030 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 18:58:12.216420 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":170,"threads_started":1}
I20260812 18:58:12.217913 8241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:12.218811 8248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a88f89dc9d544849c18172bf80b858e. Latest consensus state: current_term: 1 leader_uuid: "9a88f89dc9d544849c18172bf80b858e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260812 18:58:12.218986 8248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:12.219225 8252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a88f89dc9d544849c18172bf80b858e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88f89dc9d544849c18172bf80b858e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260812 18:58:12.219400 8252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:12.219937 8259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:12.221448 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:12.223611 8259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:12.230290 8259 catalog_manager.cc:1347] Generated new cluster ID: 6571bab2e54049368b45c3c5da47f76f
I20260812 18:58:12.230468 8259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:12.234398 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:12.223816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40141 (local address 127.0.0.1:37073)
0812 18:58:12.223961 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:12.223965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:12.225078 (+ 1113us) server_negotiation.cc:407] Connection header received
0812 18:58:12.225216 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:12.225219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:12.225277 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:12.225371 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:12.226348 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:12.227139 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:12.228471 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:12.228815 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:12.228944 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:12.230720 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:12.230724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:12.230727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:12.230755 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:12.232501 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:12.233079 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:12.233083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:12.233085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:12.233171 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:12.233482 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:12.233485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:12.233487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:12.233697 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:12.233815 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:12.234102 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 18:58:12.234231 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 18:58:12.253065 8259 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:12.254472 8259 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:12.271045 8259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a88f89dc9d544849c18172bf80b858e: Generated new TSK 0
I20260812 18:58:12.272079 8259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:12.284567 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8155
2026-08-12T18:58:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1614 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:12 dist-test-slave-rnrw krb5kdc[8279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:12 dist-test-slave-rnrw krb5kdc[8279](info): set up 2 sockets
Aug 12 18:58:12 dist-test-slave-rnrw krb5kdc[8279](info): commencing operation
krb5kdc: starting...
W20260812 18:58:13.433089 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:13 dist-test-slave-rnrw krb5kdc[8279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:13.581202 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:13.704941 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:13.705338 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:13.705435 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:13.709827 8297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:13.709957 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:13.710000 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:13.710041 8297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:13.710078 8297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:13.715943 8297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:13.717360 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:13.718569 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:13.728924 8303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:13.728868 8302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:13.728868 8305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:13.728916 8297 server_base.cc:1034] running on GCE node
I20260812 18:58:13.730016 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:13.731371 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:13.732630 8297 hybrid_clock.cc:648] HybridClock initialized: now 1786561093732610 us; error 38 us; skew 500 ppm
Aug 12 18:58:13 dist-test-slave-rnrw krb5kdc[8279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:13.742851 8297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:13.744082 8297 webserver.cc:460] Webserver started at http://127.0.0.1:46813/ using document root <none> and password file <none>
I20260812 18:58:13.744814 8297 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:13.744886 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:13.745154 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:13.747424 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "35f0ed3b8a11471a9afb35999ee842ee"
format_stamp: "Formatted at 2026-08-12 18:58:13 on dist-test-slave-rnrw"
server_key: "a9f0f0fc017928a2d9db8e221eff512f"
server_key_iv: "46dcbd4991a3962cc14af7248b7b28f2"
server_key_version: "encryptionkey@0"
I20260812 18:58:13.748039 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "35f0ed3b8a11471a9afb35999ee842ee"
format_stamp: "Formatted at 2026-08-12 18:58:13 on dist-test-slave-rnrw"
server_key: "a9f0f0fc017928a2d9db8e221eff512f"
server_key_iv: "46dcbd4991a3962cc14af7248b7b28f2"
server_key_version: "encryptionkey@0"
I20260812 18:58:13.752228 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:13.755127 8314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:13.756438 8297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 18:58:13.756681 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "35f0ed3b8a11471a9afb35999ee842ee"
format_stamp: "Formatted at 2026-08-12 18:58:13 on dist-test-slave-rnrw"
server_key: "a9f0f0fc017928a2d9db8e221eff512f"
server_key_iv: "46dcbd4991a3962cc14af7248b7b28f2"
server_key_version: "encryptionkey@0"
I20260812 18:58:13.756799 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:13.770473 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:13.774327 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:13.774572 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:13.783088 8297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34689
I20260812 18:58:13.783133 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34689 every 8 connection(s)
I20260812 18:58:13.784214 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:13.787590 8377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:13.790592 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8297
I20260812 18:58:13.790741 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:13.791075 16547 external_mini_cluster.cc:1428] Setting key 83dadad62b530288f3f1a40834d57b05
I20260812 18:58:13.794968 8377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee: Bootstrap starting.
I20260812 18:58:13.797966 8377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:13.798954 8377 log.cc:824] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:13.802095 8377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee: No bootstrap required, opened a new log
I20260812 18:58:13.803038 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:13.792895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60351 (local address 127.0.0.1:34689)
0812 18:58:13.793424 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:13.793447 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:13.793498 (+ 51us) server_negotiation.cc:407] Connection header received
0812 18:58:13.794409 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:13.794434 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:13.794899 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:13.795314 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:13.795772 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:13.797345 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:13.798457 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:13.799192 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:13.799379 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:13.800709 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:13.800716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:13.800737 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:13.801323 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:13.801405 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:13.801437 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:13.801556 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:13.801752 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 18:58:13.802022 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":164,"threads_started":1}
I20260812 18:58:13.806151 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:13.806238 8377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260812 18:58:13.806519 8377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:13.806602 8377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f0ed3b8a11471a9afb35999ee842ee, State: Initialized, Role: FOLLOWER
I20260812 18:58:13.807061 8377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260812 18:58:13.807194 8377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:13.807284 8377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:13.807425 8377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:13.808450 8377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260812 18:58:13.808964 8377 leader_election.cc:302] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35f0ed3b8a11471a9afb35999ee842ee; no voters:
I20260812 18:58:13.809361 8377 leader_election.cc:288] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:13.809482 8389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:13.809783 8389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [term 1 LEADER]: Becoming Leader. State: Replica: 35f0ed3b8a11471a9afb35999ee842ee, State: Running, Role: LEADER
I20260812 18:58:13.810154 8389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } }
I20260812 18:58:13.810896 8377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:13.812291 8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f0ed3b8a11471a9afb35999ee842ee. Latest consensus state: current_term: 1 leader_uuid: "35f0ed3b8a11471a9afb35999ee842ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } } }
I20260812 18:58:13.812438 8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:13.812923 8390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35f0ed3b8a11471a9afb35999ee842ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f0ed3b8a11471a9afb35999ee842ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34689 } } }
I20260812 18:58:13.813088 8390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:13.813447 8401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:13.813933 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:13.808696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41773 (local address 127.0.0.1:34689)
0812 18:58:13.808830 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:13.808834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:13.809009 (+ 175us) server_negotiation.cc:407] Connection header received
0812 18:58:13.809217 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:13.809220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:13.809306 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:13.809415 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:13.809832 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:13.810602 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:13.811718 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:13.812105 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:13.812238 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:13.813404 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:13.813408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:13.813411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:13.813426 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:13.813457 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:13.813476 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:13.813533 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:13.813734 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 18:58:13.813800 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:58:13.817260 8401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:13.823879 8401 catalog_manager.cc:1347] Generated new cluster ID: ee688d16fd78408585d87c18e58f8d81
I20260812 18:58:13.823977 8401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:13.841815 8401 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:13.843252 8401 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:13.853616 8401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35f0ed3b8a11471a9afb35999ee842ee: Generated new TSK 0
I20260812 18:58:13.854465 8401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:13.863253 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8297
2026-08-12T18:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:13 dist-test-slave-rnrw krb5kdc[8413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:13 dist-test-slave-rnrw krb5kdc[8413](info): set up 2 sockets
Aug 12 18:58:13 dist-test-slave-rnrw krb5kdc[8413](info): commencing operation
krb5kdc: starting...
W20260812 18:58:15.003290 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:15 dist-test-slave-rnrw krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:15.144459 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:15.267320 8428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:15.267795 8428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:15.267891 8428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:15.272225 8428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:15.272353 8428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:15.272394 8428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:15.272433 8428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:15.272495 8428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:15.278396 8428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:15.279707 8428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:15.280898 8428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:15.290421 8434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:15.290431 8433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:15.290580 8436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:15.290916 8428 server_base.cc:1034] running on GCE node
I20260812 18:58:15.291572 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:15.292898 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:15.294112 8428 hybrid_clock.cc:648] HybridClock initialized: now 1786561095294077 us; error 51 us; skew 500 ppm
Aug 12 18:58:15 dist-test-slave-rnrw krb5kdc[8413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:15.297485 8428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:15.298655 8428 webserver.cc:460] Webserver started at http://127.0.0.1:40473/ using document root <none> and password file <none>
I20260812 18:58:15.299302 8428 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:15.299402 8428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:15.299683 8428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:15.302021 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "75de17a2f94543fb8b49c5e373979783"
format_stamp: "Formatted at 2026-08-12 18:58:15 on dist-test-slave-rnrw"
server_key: "3a05273613c3e8f9fa8d3de0cb0c0acc"
server_key_iv: "ffebbb12b7f77116dd336d1a32acb56f"
server_key_version: "encryptionkey@0"
I20260812 18:58:15.302690 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "75de17a2f94543fb8b49c5e373979783"
format_stamp: "Formatted at 2026-08-12 18:58:15 on dist-test-slave-rnrw"
server_key: "3a05273613c3e8f9fa8d3de0cb0c0acc"
server_key_iv: "ffebbb12b7f77116dd336d1a32acb56f"
server_key_version: "encryptionkey@0"
I20260812 18:58:15.307024 8428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:15.310029 8445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:15.311513 8428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:58:15.311703 8428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "75de17a2f94543fb8b49c5e373979783"
format_stamp: "Formatted at 2026-08-12 18:58:15 on dist-test-slave-rnrw"
server_key: "3a05273613c3e8f9fa8d3de0cb0c0acc"
server_key_iv: "ffebbb12b7f77116dd336d1a32acb56f"
server_key_version: "encryptionkey@0"
I20260812 18:58:15.311844 8428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:15.317065 8428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:15.320436 8428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:15.320698 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:15.328693 8428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44003
I20260812 18:58:15.328756 8503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44003 every 8 connection(s)
I20260812 18:58:15.329748 8428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:15.332890 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8428
I20260812 18:58:15.333078 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:15.333169 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:15.333434 16547 external_mini_cluster.cc:1428] Setting key 102f0d1c39e9c2d3d0a717cae12620e6
I20260812 18:58:15.339368 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:15.334956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39983 (local address 127.0.0.1:44003)
0812 18:58:15.335450 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:15.335467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:15.335507 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:58:15.336324 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:15.336344 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:15.336733 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:15.337039 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:15.338020 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:15.338026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:15.338039 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:15.338051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:15.338109 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:15.338129 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:15.338247 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:15.338366 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 18:58:15.338546 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":187,"threads_started":1}
I20260812 18:58:15.341380 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783: Bootstrap starting.
I20260812 18:58:15.342497 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:15.344422 8504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:15.345588 8504 log.cc:824] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:15.347433 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:15.345331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37275 (local address 127.0.0.1:44003)
0812 18:58:15.345493 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:15.345499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:15.345798 (+ 299us) server_negotiation.cc:407] Connection header received
0812 18:58:15.345953 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:15.345955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:15.345995 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:15.346066 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:15.346957 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:15.346961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:15.346964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:15.346979 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:15.347012 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:15.347026 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:15.347140 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:15.347240 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 18:58:15.347286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 18:58:15.348312 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783: No bootstrap required, opened a new log
I20260812 18:58:15.352439 8504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260812 18:58:15.352825 8504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:15.352936 8504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75de17a2f94543fb8b49c5e373979783, State: Initialized, Role: FOLLOWER
I20260812 18:58:15.353500 8504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260812 18:58:15.353703 8504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:15.353790 8504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:15.353910 8504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:15.355024 8504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260812 18:58:15.355429 8504 leader_election.cc:302] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75de17a2f94543fb8b49c5e373979783; no voters:
I20260812 18:58:15.355849 8504 leader_election.cc:288] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:15.356015 8517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:15.356315 8517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [term 1 LEADER]: Becoming Leader. State: Replica: 75de17a2f94543fb8b49c5e373979783, State: Running, Role: LEADER
I20260812 18:58:15.356734 8517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260812 18:58:15.357376 8504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:15.358673 8518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75de17a2f94543fb8b49c5e373979783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } } }
I20260812 18:58:15.358832 8518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:15.359162 8520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75de17a2f94543fb8b49c5e373979783. Latest consensus state: current_term: 1 leader_uuid: "75de17a2f94543fb8b49c5e373979783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75de17a2f94543fb8b49c5e373979783" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } } }
I20260812 18:58:15.359272 8520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:15.359273 8532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:15.363210 8532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:15.370764 8532 catalog_manager.cc:1347] Generated new cluster ID: c9143788e3af4451a79132fa4dc42089
I20260812 18:58:15.370878 8532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:15.380862 8532 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:15.381304 8532 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:15.393081 8532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75de17a2f94543fb8b49c5e373979783: Generated new TSK 0
I20260812 18:58:15.394026 8532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:15.399544 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8428
2026-08-12T18:58:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1537 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:15 dist-test-slave-rnrw krb5kdc[8548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:15 dist-test-slave-rnrw krb5kdc[8548](info): set up 2 sockets
Aug 12 18:58:15 dist-test-slave-rnrw krb5kdc[8548](info): commencing operation
krb5kdc: starting...
W20260812 18:58:16.560913 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:16 dist-test-slave-rnrw krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:16.705506 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:16.834383 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:16.834702 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:16.834748 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:16.839165 8562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:16.839265 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:16.839291 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:16.839309 8562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:16.839325 8562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:16.845396 8562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:16.846625 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:16.847716 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:16.857088 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:16.857164 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:16.857198 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:16.857385 8562 server_base.cc:1034] running on GCE node
I20260812 18:58:16.858089 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:16.859378 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:16.860625 8562 hybrid_clock.cc:648] HybridClock initialized: now 1786561096860609 us; error 29 us; skew 500 ppm
Aug 12 18:58:16 dist-test-slave-rnrw krb5kdc[8548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:16.863741 8562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:16.864835 8562 webserver.cc:460] Webserver started at http://127.0.0.1:36903/ using document root <none> and password file <none>
I20260812 18:58:16.865449 8562 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:16.865546 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:16.865793 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:16.868006 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "16b754c45efe409ea1e7aaf1f842e2b7"
format_stamp: "Formatted at 2026-08-12 18:58:16 on dist-test-slave-rnrw"
server_key: "b5f3e35bf5f0e84cca58c80caed034ca"
server_key_iv: "fa6d163b65c81b352a0e706832bc14c2"
server_key_version: "encryptionkey@0"
I20260812 18:58:16.868676 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "16b754c45efe409ea1e7aaf1f842e2b7"
format_stamp: "Formatted at 2026-08-12 18:58:16 on dist-test-slave-rnrw"
server_key: "b5f3e35bf5f0e84cca58c80caed034ca"
server_key_iv: "fa6d163b65c81b352a0e706832bc14c2"
server_key_version: "encryptionkey@0"
I20260812 18:58:16.872987 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:58:16.876235 8578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:16.877877 8562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:16.878100 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "16b754c45efe409ea1e7aaf1f842e2b7"
format_stamp: "Formatted at 2026-08-12 18:58:16 on dist-test-slave-rnrw"
server_key: "b5f3e35bf5f0e84cca58c80caed034ca"
server_key_iv: "fa6d163b65c81b352a0e706832bc14c2"
server_key_version: "encryptionkey@0"
I20260812 18:58:16.878361 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:16.899073 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:16.902907 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:16.903152 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:16.911300 8562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42189
I20260812 18:58:16.911332 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42189 every 8 connection(s)
I20260812 18:58:16.912456 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:16.915069 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8562
I20260812 18:58:16.915198 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:16.915558 16547 external_mini_cluster.cc:1428] Setting key 9fd9c971dfdac266e072e22684fa1ee0
I20260812 18:58:16.915884 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:16.922956 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: Bootstrap starting.
Aug 12 18:58:16 dist-test-slave-rnrw krb5kdc[8548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 18:58:16.925693 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:16.926505 8640 log.cc:824] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:16.928719 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: No bootstrap required, opened a new log
I20260812 18:58:16.931193 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:16.917189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:49049 (local address 127.0.0.1:42189)
0812 18:58:16.917729 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:16.917746 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:16.917787 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:58:16.918546 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:16.918568 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:16.918951 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:16.919342 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:16.920235 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:16.921368 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:16.922311 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:16.922838 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:16.923019 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:16.926035 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:16.926042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:16.926058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:16.926091 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:16.927960 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:16.928595 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:16.928602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:16.928615 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:16.928731 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:16.929097 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:16.929107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:16.929109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:16.929540 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:16.929795 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:16.930128 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 18:58:16.930373 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":180,"threads_started":1}
I20260812 18:58:16.932744 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } }
I20260812 18:58:16.933037 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:16.933090 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b754c45efe409ea1e7aaf1f842e2b7, State: Initialized, Role: FOLLOWER
I20260812 18:58:16.933578 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } }
I20260812 18:58:16.933765 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:16.933851 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:16.933979 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:16.934648 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:16.935168 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } }
I20260812 18:58:16.935531 8640 leader_election.cc:302] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b754c45efe409ea1e7aaf1f842e2b7; no voters:
I20260812 18:58:16.935935 8640 leader_election.cc:288] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:16.935992 8645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:16.936209 8645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [term 1 LEADER]: Becoming Leader. State: Replica: 16b754c45efe409ea1e7aaf1f842e2b7, State: Running, Role: LEADER
I20260812 18:58:16.936520 8645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } }
I20260812 18:58:16.937394 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:16.938467 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b754c45efe409ea1e7aaf1f842e2b7. Latest consensus state: current_term: 1 leader_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } } }
I20260812 18:58:16.938638 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:16.939110 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b754c45efe409ea1e7aaf1f842e2b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42189 } } }
I20260812 18:58:16.939257 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7 [sys.catalog]: This master's current role is: LEADER
W20260812 18:58:16.943269 8667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:16.943361 8667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:16.943432 8659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:16.944358 8659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:16.948376 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:16.937395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:36061 (local address 127.0.0.1:42189)
0812 18:58:16.937546 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:16.937551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:16.937994 (+ 443us) server_negotiation.cc:407] Connection header received
0812 18:58:16.938216 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:16.938221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:16.938286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:16.938368 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:16.939505 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:16.940239 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:16.941375 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:16.941711 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:16.941818 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:16.943679 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:16.943683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:16.943686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:16.943714 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:16.945612 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:16.946285 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:16.946290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:16.946294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:16.946418 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:16.946892 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:16.946907 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:16.946909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:16.947159 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:16.947311 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:16.948088 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 18:58:16.948204 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 18:58:16.951020 8659 catalog_manager.cc:1347] Generated new cluster ID: 5145e14618d14697be7986b1e546f742
I20260812 18:58:16.951119 8659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:16.968742 8659 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:16.969872 8659 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:16.980268 8659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b754c45efe409ea1e7aaf1f842e2b7: Generated new TSK 0
I20260812 18:58:16.981187 8659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:16.988615 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8562
W20260812 18:58:16.999387 8545 connection.cc:535] client connection to 127.0.0.1:42189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42189): Connection reset by peer (error 104)
2026-08-12T18:58:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1589 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:17 dist-test-slave-rnrw krb5kdc[8673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:17 dist-test-slave-rnrw krb5kdc[8673](info): set up 2 sockets
Aug 12 18:58:17 dist-test-slave-rnrw krb5kdc[8673](info): commencing operation
krb5kdc: starting...
W20260812 18:58:18.127741 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:18 dist-test-slave-rnrw krb5kdc[8673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:18.276939 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:18.399467 8689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:18.399837 8689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:18.399927 8689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:18.404347 8689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:18.404461 8689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:18.404502 8689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:18.404585 8689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:18.404637 8689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:18.410449 8689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:18.411726 8689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:18.412874 8689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:58:18.422361 8689 server_base.cc:1034] running on GCE node
W20260812 18:58:18.422299 8696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:18.422303 8695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:18.422299 8698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:18.423321 8689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:18.424537 8689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:18.425818 8689 hybrid_clock.cc:648] HybridClock initialized: now 1786561098425795 us; error 37 us; skew 500 ppm
Aug 12 18:58:18 dist-test-slave-rnrw krb5kdc[8673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:18.429076 8689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:18.430178 8689 webserver.cc:460] Webserver started at http://127.0.0.1:40721/ using document root <none> and password file <none>
I20260812 18:58:18.430806 8689 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:18.430902 8689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:18.431151 8689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:18.433481 8689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "2d116e020c1e4f56954904d5b24d8d58"
format_stamp: "Formatted at 2026-08-12 18:58:18 on dist-test-slave-rnrw"
server_key: "b546bf991f0a670ac6a49d42f997f924"
server_key_iv: "003e3372951efeb584886a5769ee6f0a"
server_key_version: "encryptionkey@0"
I20260812 18:58:18.434124 8689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "2d116e020c1e4f56954904d5b24d8d58"
format_stamp: "Formatted at 2026-08-12 18:58:18 on dist-test-slave-rnrw"
server_key: "b546bf991f0a670ac6a49d42f997f924"
server_key_iv: "003e3372951efeb584886a5769ee6f0a"
server_key_version: "encryptionkey@0"
I20260812 18:58:18.438549 8689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:18.441570 8708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:18.443075 8689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:18.443257 8689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "2d116e020c1e4f56954904d5b24d8d58"
format_stamp: "Formatted at 2026-08-12 18:58:18 on dist-test-slave-rnrw"
server_key: "b546bf991f0a670ac6a49d42f997f924"
server_key_iv: "003e3372951efeb584886a5769ee6f0a"
server_key_version: "encryptionkey@0"
I20260812 18:58:18.443397 8689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:18.457906 8689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:18.461540 8689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:18.461783 8689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:18.470428 8689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35691
I20260812 18:58:18.470458 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35691 every 8 connection(s)
I20260812 18:58:18.471508 8689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:18.474531 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:18.475481 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8689
I20260812 18:58:18.475620 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:18.475970 16547 external_mini_cluster.cc:1428] Setting key 9f6c95b335204d20ec8eb768d3bdd30e
I20260812 18:58:18.481259 8770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58: Bootstrap starting.
I20260812 18:58:18.483770 8770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:18.484647 8770 log.cc:824] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:18.485910 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:18.477586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:33295 (local address 127.0.0.1:35691)
0812 18:58:18.478111 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:18.478129 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:18.478168 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:58:18.478818 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:18.478837 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:18.479209 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:18.479666 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:18.480099 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:18.481170 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:18.482135 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:18.482630 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:18.482814 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:18.483826 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:18.483831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:18.483845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:18.484315 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:18.484376 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:18.484400 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:18.484556 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:18.484774 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 18:58:18.484935 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":163,"threads_started":1}
I20260812 18:58:18.487149 8770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58: No bootstrap required, opened a new log
I20260812 18:58:18.489406 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:18.491281 8770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260812 18:58:18.491556 8770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:18.491639 8770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d116e020c1e4f56954904d5b24d8d58, State: Initialized, Role: FOLLOWER
I20260812 18:58:18.492085 8770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260812 18:58:18.492245 8770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:18.492380 8770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:18.492503 8770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:18.493691 8770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260812 18:58:18.494109 8770 leader_election.cc:302] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d116e020c1e4f56954904d5b24d8d58; no voters:
I20260812 18:58:18.494547 8770 leader_election.cc:288] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:18.494628 8783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:18.494867 8783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [term 1 LEADER]: Becoming Leader. State: Replica: 2d116e020c1e4f56954904d5b24d8d58, State: Running, Role: LEADER
I20260812 18:58:18.495378 8783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260812 18:58:18.496241 8770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:18.496526 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:18.491664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:60457 (local address 127.0.0.1:35691)
0812 18:58:18.491796 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:18.491802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:18.491984 (+ 182us) server_negotiation.cc:407] Connection header received
0812 18:58:18.492156 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:18.492165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:18.492211 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:18.492302 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:18.492683 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:18.493201 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:18.494365 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:18.494664 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:18.494847 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:18.495832 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:18.495836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:18.495838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:18.495853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:18.495885 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:18.495897 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:18.496313 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:18.496374 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 18:58:18.496417 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 18:58:18.496793 8784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d116e020c1e4f56954904d5b24d8d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260812 18:58:18.496918 8784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:18.497641 8785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d116e020c1e4f56954904d5b24d8d58. Latest consensus state: current_term: 1 leader_uuid: "2d116e020c1e4f56954904d5b24d8d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d116e020c1e4f56954904d5b24d8d58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260812 18:58:18.497732 8785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:18.498153 8787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:18.501618 8787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:18.508621 8787 catalog_manager.cc:1347] Generated new cluster ID: 98f32831933946ee929d09e191fe040d
I20260812 18:58:18.508718 8787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:18.534663 8787 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:18.535748 8787 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:18.541720 8787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d116e020c1e4f56954904d5b24d8d58: Generated new TSK 0
I20260812 18:58:18.542623 8787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:18.546711 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8689
W20260812 18:58:18.558090 8671 connection.cc:535] client connection to 127.0.0.1:35691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35691): Connection reset by peer (error 104)
2026-08-12T18:58:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:18 dist-test-slave-rnrw krb5kdc[8804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:18 dist-test-slave-rnrw krb5kdc[8804](info): set up 2 sockets
Aug 12 18:58:18 dist-test-slave-rnrw krb5kdc[8804](info): commencing operation
krb5kdc: starting...
W20260812 18:58:19.683094 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:19 dist-test-slave-rnrw krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:19.832974 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:19.953703 8819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:19.954094 8819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:19.954181 8819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:19.958633 8819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:19.958770 8819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:19.958809 8819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:19.958847 8819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:19.958884 8819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:19.964676 8819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:19.966001 8819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:19.967131 8819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:19.977005 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:19.977258 8819 server_base.cc:1034] running on GCE node
W20260812 18:58:19.977005 8828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:19.977023 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:19.978062 8819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:19.979354 8819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:19.980628 8819 hybrid_clock.cc:648] HybridClock initialized: now 1786561099980599 us; error 34 us; skew 500 ppm
Aug 12 18:58:19 dist-test-slave-rnrw krb5kdc[8804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:19.990257 8819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:19.991451 8819 webserver.cc:460] Webserver started at http://127.0.0.1:37637/ using document root <none> and password file <none>
I20260812 18:58:19.992076 8819 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:19.992143 8819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:19.992331 8819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:19.994656 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "f2cdb933d217471d8b75506771acda41"
format_stamp: "Formatted at 2026-08-12 18:58:19 on dist-test-slave-rnrw"
server_key: "5722edb33601955a92b8e0d7d8b8a80a"
server_key_iv: "667d5bd4386ca75f84a8f17938a8cc04"
server_key_version: "encryptionkey@0"
I20260812 18:58:19.995245 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "f2cdb933d217471d8b75506771acda41"
format_stamp: "Formatted at 2026-08-12 18:58:19 on dist-test-slave-rnrw"
server_key: "5722edb33601955a92b8e0d7d8b8a80a"
server_key_iv: "667d5bd4386ca75f84a8f17938a8cc04"
server_key_version: "encryptionkey@0"
I20260812 18:58:19.999591 8819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:58:20.002622 8838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:20.004171 8819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:58:20.004348 8819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "f2cdb933d217471d8b75506771acda41"
format_stamp: "Formatted at 2026-08-12 18:58:19 on dist-test-slave-rnrw"
server_key: "5722edb33601955a92b8e0d7d8b8a80a"
server_key_iv: "667d5bd4386ca75f84a8f17938a8cc04"
server_key_version: "encryptionkey@0"
I20260812 18:58:20.004492 8819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:20.010107 8819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:20.013919 8819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:20.014160 8819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:20.022600 8819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45317
I20260812 18:58:20.022605 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45317 every 8 connection(s)
I20260812 18:58:20.023728 8819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:20.027084 8902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:20.031576 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8819
I20260812 18:58:20.031731 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:20.032107 16547 external_mini_cluster.cc:1428] Setting key 7d08c7991c2bbf70b892cafdf2928220
I20260812 18:58:20.034319 8902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41: Bootstrap starting.
I20260812 18:58:20.037006 8902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:20.037985 8902 log.cc:824] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:20.038153 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:20.033803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:55923 (local address 127.0.0.1:45317)
0812 18:58:20.034410 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:20.034434 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:20.034485 (+ 51us) server_negotiation.cc:407] Connection header received
0812 18:58:20.035206 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:20.035225 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:20.035536 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:20.035899 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:20.036814 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:20.036822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:20.036841 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:20.036852 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:20.036912 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:20.036932 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:20.037072 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:20.037178 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 18:58:20.037364 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":163,"threads_started":1}
I20260812 18:58:20.040448 8902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41: No bootstrap required, opened a new log
I20260812 18:58:20.041549 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:20.044657 8902 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260812 18:58:20.045009 8902 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:20.045106 8902 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdb933d217471d8b75506771acda41, State: Initialized, Role: FOLLOWER
I20260812 18:58:20.045639 8902 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260812 18:58:20.045847 8902 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:20.045851 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:20.043737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:57729 (local address 127.0.0.1:45317)
0812 18:58:20.043881 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:20.043888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:20.044096 (+ 208us) server_negotiation.cc:407] Connection header received
0812 18:58:20.044249 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:20.044252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:20.044290 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:20.044384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:20.045245 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:20.045248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:20.045251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:20.045263 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:20.045299 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:20.045311 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:20.045422 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:20.045575 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 18:58:20.045613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 18:58:20.045972 8902 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:20.046135 8902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:20.047292 8902 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260812 18:58:20.047638 8902 leader_election.cc:302] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2cdb933d217471d8b75506771acda41; no voters:
I20260812 18:58:20.047963 8902 leader_election.cc:288] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:20.048045 8915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:20.048259 8915 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [term 1 LEADER]: Becoming Leader. State: Replica: f2cdb933d217471d8b75506771acda41, State: Running, Role: LEADER
I20260812 18:58:20.048606 8915 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260812 18:58:20.049522 8902 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:20.050436 8917 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2cdb933d217471d8b75506771acda41. Latest consensus state: current_term: 1 leader_uuid: "f2cdb933d217471d8b75506771acda41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } } }
I20260812 18:58:20.050592 8917 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:20.050983 8918 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2cdb933d217471d8b75506771acda41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2cdb933d217471d8b75506771acda41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } } }
I20260812 18:58:20.051116 8918 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:20.052196 8924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:20.055143 8924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:20.060951 8924 catalog_manager.cc:1347] Generated new cluster ID: 2ba081863004432bbe5a086b67772d01
I20260812 18:58:20.061049 8924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:20.099027 8924 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:20.099404 8924 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:20.103267 8924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2cdb933d217471d8b75506771acda41: Generated new TSK 0
I20260812 18:58:20.104039 8924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:20.114652 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8819
2026-08-12T18:58:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1566 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:20 dist-test-slave-rnrw krb5kdc[8939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:20 dist-test-slave-rnrw krb5kdc[8939](info): set up 2 sockets
Aug 12 18:58:20 dist-test-slave-rnrw krb5kdc[8939](info): commencing operation
krb5kdc: starting...
W20260812 18:58:21.246906 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:21.389341 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:21.519927 8952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:21.520308 8952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:21.520401 8952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:21.524725 8952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:21.524859 8952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:21.524897 8952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:21.524936 8952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:21.524974 8952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:21.532219 8952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:21.533501 8952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:21.534616 8952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:21.544255 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:21.544293 8961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:21.544260 8963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:21.544672 8952 server_base.cc:1034] running on GCE node
I20260812 18:58:21.545497 8952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:21.546870 8952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:21.548098 8952 hybrid_clock.cc:648] HybridClock initialized: now 1786561101548065 us; error 57 us; skew 500 ppm
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[8939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:21.551540 8952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:21.552745 8952 webserver.cc:460] Webserver started at http://127.0.0.1:39681/ using document root <none> and password file <none>
I20260812 18:58:21.553390 8952 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:21.553462 8952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:21.553663 8952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:21.556079 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "a396682b93ed4420aa681bba0087fdff"
format_stamp: "Formatted at 2026-08-12 18:58:21 on dist-test-slave-rnrw"
server_key: "f4117d194b46ace809dcd7881f032385"
server_key_iv: "a14071881af234842eb22cbb92fec50f"
server_key_version: "encryptionkey@0"
I20260812 18:58:21.556772 8952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "a396682b93ed4420aa681bba0087fdff"
format_stamp: "Formatted at 2026-08-12 18:58:21 on dist-test-slave-rnrw"
server_key: "f4117d194b46ace809dcd7881f032385"
server_key_iv: "a14071881af234842eb22cbb92fec50f"
server_key_version: "encryptionkey@0"
I20260812 18:58:21.561271 8952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:58:21.564195 8972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:21.565730 8952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:21.565852 8952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "a396682b93ed4420aa681bba0087fdff"
format_stamp: "Formatted at 2026-08-12 18:58:21 on dist-test-slave-rnrw"
server_key: "f4117d194b46ace809dcd7881f032385"
server_key_iv: "a14071881af234842eb22cbb92fec50f"
server_key_version: "encryptionkey@0"
I20260812 18:58:21.565955 8952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:21.585224 8952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:21.588902 8952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:21.589085 8952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:21.597383 8952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39315
I20260812 18:58:21.597406 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39315 every 8 connection(s)
I20260812 18:58:21.598433 8952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:21.601719 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:21.608057 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff: Bootstrap starting.
I20260812 18:58:21.608467 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 8952
I20260812 18:58:21.608639 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:21.608959 16547 external_mini_cluster.cc:1428] Setting key de3b5733616c86c223f6fda2352909af
I20260812 18:58:21.611279 9034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:21.612190 9034 log.cc:824] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:21.614848 9034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff: No bootstrap required, opened a new log
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[8939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 18:58:21.620296 9034 raft_consensus.cc:348] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260812 18:58:21.620644 9034 raft_consensus.cc:374] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:21.620725 9034 raft_consensus.cc:729] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a396682b93ed4420aa681bba0087fdff, State: Initialized, Role: FOLLOWER
I20260812 18:58:21.621325 9034 consensus_queue.cc:260] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260812 18:58:21.621539 9034 raft_consensus.cc:388] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:21.621656 9034 raft_consensus.cc:482] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:21.621799 9034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:21.622953 9034 raft_consensus.cc:504] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260812 18:58:21.623383 9034 leader_election.cc:302] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a396682b93ed4420aa681bba0087fdff; no voters:
I20260812 18:58:21.623842 9034 leader_election.cc:288] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:21.623931 9041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:21.624181 9041 raft_consensus.cc:686] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [term 1 LEADER]: Becoming Leader. State: Replica: a396682b93ed4420aa681bba0087fdff, State: Running, Role: LEADER
I20260812 18:58:21.624634 9041 consensus_queue.cc:237] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } }
I20260812 18:58:21.625432 9034 sys_catalog.cc:564] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:21.626453 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a396682b93ed4420aa681bba0087fdff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } } }
I20260812 18:58:21.626713 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:21.626945 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:21.610630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:49663 (local address 127.0.0.1:39315)
0812 18:58:21.611164 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:21.611180 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:21.611219 (+ 39us) server_negotiation.cc:407] Connection header received
0812 18:58:21.612097 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:21.612134 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:21.612640 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:21.613055 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:21.614054 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:21.615168 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:21.616474 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:21.617082 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:21.617214 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:21.620712 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:21.620717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:21.620734 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:21.620800 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:21.622889 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:21.623507 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:21.623514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:21.623523 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:21.623676 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:21.624125 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:21.624128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:21.624130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:21.624773 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:21.625038 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:21.625416 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 18:58:21.625907 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260812 18:58:21.627044 9042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a396682b93ed4420aa681bba0087fdff. Latest consensus state: current_term: 1 leader_uuid: "a396682b93ed4420aa681bba0087fdff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a396682b93ed4420aa681bba0087fdff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39315 } } }
I20260812 18:58:21.627286 9042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:21.628342 9051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:21.630991 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:21.631909 9051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:21.638437 9051 catalog_manager.cc:1347] Generated new cluster ID: cb529ebfe30c4109bf8653084f01667f
I20260812 18:58:21.638587 9051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:21.646834 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:21.636522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:40679 (local address 127.0.0.1:39315)
0812 18:58:21.636692 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:21.636700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:21.637157 (+ 457us) server_negotiation.cc:407] Connection header received
0812 18:58:21.637329 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:21.637333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:21.637398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:21.637506 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:21.638484 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:21.639312 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:21.640622 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:21.641011 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:21.641135 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:21.643025 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:21.643029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:21.643032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:21.643061 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:21.644831 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:21.645436 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:21.645439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:21.645441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:21.645528 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:21.645893 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:21.645897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:21.645898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:21.646120 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:21.646258 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:21.646558 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 18:58:21.646663 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 18:58:21.651690 9051 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:21.653090 9051 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:21.661141 9051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a396682b93ed4420aa681bba0087fdff: Generated new TSK 0
I20260812 18:58:21.662129 9051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:21.667495 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 8952
W20260812 18:58:21.679119 8937 connection.cc:535] client connection to 127.0.0.1:39315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39315): Connection reset by peer (error 104)
2026-08-12T18:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[9073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[9073](info): set up 2 sockets
Aug 12 18:58:21 dist-test-slave-rnrw krb5kdc[9073](info): commencing operation
krb5kdc: starting...
W20260812 18:58:22.826287 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:22 dist-test-slave-rnrw krb5kdc[9073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:22.971798 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:23.092324 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:23.092756 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:23.092854 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:23.097072 9086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:23.097215 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:23.097256 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:23.097295 9086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:23.097330 9086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:23.103148 9086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:23.104440 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:23.105577 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:23.114701 9097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:23.114693 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:23.115094 9086 server_base.cc:1034] running on GCE node
W20260812 18:58:23.114706 9092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:23.115917 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:23.117244 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:23.118422 9086 hybrid_clock.cc:648] HybridClock initialized: now 1786561103118387 us; error 49 us; skew 500 ppm
Aug 12 18:58:23 dist-test-slave-rnrw krb5kdc[9073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:23.121702 9086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:23.122764 9086 webserver.cc:460] Webserver started at http://127.0.0.1:39417/ using document root <none> and password file <none>
I20260812 18:58:23.123435 9086 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:23.123549 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:23.123829 9086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:23.126171 9086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "439c3d5887ac4511b94dc1aaf2db5dd7"
format_stamp: "Formatted at 2026-08-12 18:58:23 on dist-test-slave-rnrw"
server_key: "3bb2150cac52fccad99c21403d71f941"
server_key_iv: "138edb59cc7fd62bc73e9b7cba92087f"
server_key_version: "encryptionkey@0"
I20260812 18:58:23.126796 9086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "439c3d5887ac4511b94dc1aaf2db5dd7"
format_stamp: "Formatted at 2026-08-12 18:58:23 on dist-test-slave-rnrw"
server_key: "3bb2150cac52fccad99c21403d71f941"
server_key_iv: "138edb59cc7fd62bc73e9b7cba92087f"
server_key_version: "encryptionkey@0"
I20260812 18:58:23.131049 9086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:23.133816 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:23.135128 9086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 18:58:23.135330 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "439c3d5887ac4511b94dc1aaf2db5dd7"
format_stamp: "Formatted at 2026-08-12 18:58:23 on dist-test-slave-rnrw"
server_key: "3bb2150cac52fccad99c21403d71f941"
server_key_iv: "138edb59cc7fd62bc73e9b7cba92087f"
server_key_version: "encryptionkey@0"
I20260812 18:58:23.135465 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:23.146569 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:23.149876 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:23.150126 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:23.158051 9086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46101
I20260812 18:58:23.158070 9170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46101 every 8 connection(s)
I20260812 18:58:23.159148 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:23.160323 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9086
I20260812 18:58:23.160478 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:23.160858 16547 external_mini_cluster.cc:1428] Setting key 11983f268678d6e0f3b60b6a175bd36b
I20260812 18:58:23.162942 9171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:23.169238 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7: Bootstrap starting.
I20260812 18:58:23.170681 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:23.162443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:35163 (local address 127.0.0.1:46101)
0812 18:58:23.162953 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:23.162970 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:23.163040 (+ 70us) server_negotiation.cc:407] Connection header received
0812 18:58:23.163747 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:23.163765 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:23.164140 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:23.164509 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:23.165013 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:23.166025 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:23.166984 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:23.167509 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:23.167711 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:23.168782 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:23.168787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:23.168803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:23.169223 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:23.169291 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:23.169343 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:23.169526 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:23.169728 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 18:58:23.169893 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":166,"threads_started":1}
I20260812 18:58:23.171954 9171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:23.172950 9171 log.cc:824] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:23.174281 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:23.175513 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7: No bootstrap required, opened a new log
I20260812 18:58:23.179646 9171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260812 18:58:23.179986 9171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:23.180073 9171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439c3d5887ac4511b94dc1aaf2db5dd7, State: Initialized, Role: FOLLOWER
I20260812 18:58:23.180707 9171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260812 18:58:23.180920 9171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:23.181002 9171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:23.181136 9171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:23.181803 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:23.176747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:39603 (local address 127.0.0.1:46101)
0812 18:58:23.176895 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:23.176900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:23.177108 (+ 208us) server_negotiation.cc:407] Connection header received
0812 18:58:23.177263 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:23.177266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:23.177312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:23.177384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:23.177712 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:23.178671 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:23.179815 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:23.180110 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:23.180230 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:23.181179 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:23.181182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:23.181184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:23.181218 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:23.181254 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:23.181269 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:23.181353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:23.181541 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 18:58:23.181602 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:58:23.182474 9171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260812 18:58:23.182858 9171 leader_election.cc:302] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439c3d5887ac4511b94dc1aaf2db5dd7; no voters:
I20260812 18:58:23.183246 9171 leader_election.cc:288] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:23.183379 9181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:23.183661 9181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 439c3d5887ac4511b94dc1aaf2db5dd7, State: Running, Role: LEADER
I20260812 18:58:23.184029 9181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260812 18:58:23.184686 9171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:23.185850 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439c3d5887ac4511b94dc1aaf2db5dd7. Latest consensus state: current_term: 1 leader_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260812 18:58:23.185986 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:23.186414 9195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:23.186371 9182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439c3d5887ac4511b94dc1aaf2db5dd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260812 18:58:23.186529 9182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:23.189725 9195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:23.196094 9195 catalog_manager.cc:1347] Generated new cluster ID: 5d9be84f11e24900aa9fd6b1a718d275
I20260812 18:58:23.196203 9195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:23.233575 9195 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:23.234661 9195 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:23.241523 9195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439c3d5887ac4511b94dc1aaf2db5dd7: Generated new TSK 0
I20260812 18:58:23.242365 9195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:23.249810 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9086
W20260812 18:58:23.260222 9070 connection.cc:535] client connection to 127.0.0.1:46101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46101): Connection reset by peer (error 104)
2026-08-12T18:58:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:23 dist-test-slave-rnrw krb5kdc[9212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:23 dist-test-slave-rnrw krb5kdc[9212](info): set up 2 sockets
Aug 12 18:58:23 dist-test-slave-rnrw krb5kdc[9212](info): commencing operation
krb5kdc: starting...
W20260812 18:58:24.394474 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:24 dist-test-slave-rnrw krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 18:58:24.540267 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:24.660151 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:24.660462 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:24.660507 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:24.664747 9229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:24.664832 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:24.664853 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:24.664870 9229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:24.664886 9229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:24.670554 9229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:24.671856 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:24.673055 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:24.682397 9239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:24.682400 9235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:24.682442 9236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:24.682802 9229 server_base.cc:1034] running on GCE node
I20260812 18:58:24.683475 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:24.684722 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:24.685936 9229 hybrid_clock.cc:648] HybridClock initialized: now 1786561104685914 us; error 29 us; skew 500 ppm
Aug 12 18:58:24 dist-test-slave-rnrw krb5kdc[9212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:24.689157 9229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 18:58:24.690241 9229 webserver.cc:460] Webserver started at http://127.0.0.1:45423/ using document root <none> and password file <none>
I20260812 18:58:24.690892 9229 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:24.690991 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:24.691258 9229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:24.693565 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "0faf8043c4184425a7669c4146379150"
format_stamp: "Formatted at 2026-08-12 18:58:24 on dist-test-slave-rnrw"
server_key: "0ea082e45f7f9aa20c4daa4fd95d263f"
server_key_iv: "6811f6e8d26a3cdde9dd0e2230d08787"
server_key_version: "encryptionkey@0"
I20260812 18:58:24.694186 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "0faf8043c4184425a7669c4146379150"
format_stamp: "Formatted at 2026-08-12 18:58:24 on dist-test-slave-rnrw"
server_key: "0ea082e45f7f9aa20c4daa4fd95d263f"
server_key_iv: "6811f6e8d26a3cdde9dd0e2230d08787"
server_key_version: "encryptionkey@0"
I20260812 18:58:24.699565 9229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 18:58:24.702876 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:24.704375 9229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:58:24.704628 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "0faf8043c4184425a7669c4146379150"
format_stamp: "Formatted at 2026-08-12 18:58:24 on dist-test-slave-rnrw"
server_key: "0ea082e45f7f9aa20c4daa4fd95d263f"
server_key_iv: "6811f6e8d26a3cdde9dd0e2230d08787"
server_key_version: "encryptionkey@0"
I20260812 18:58:24.704778 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:24.709956 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:24.713708 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:24.713937 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:24.722249 9229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38327
I20260812 18:58:24.722277 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38327 every 8 connection(s)
I20260812 18:58:24.723376 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:24.726545 9311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:24.729063 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9229
I20260812 18:58:24.729223 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:24.729543 16547 external_mini_cluster.cc:1428] Setting key 248aa8ce7555b08826678065f3770c15
I20260812 18:58:24.733353 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150: Bootstrap starting.
I20260812 18:58:24.735286 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:24.731149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:44013 (local address 127.0.0.1:38327)
0812 18:58:24.731677 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:24.731694 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:24.731734 (+ 40us) server_negotiation.cc:407] Connection header received
0812 18:58:24.732407 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:24.732426 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:24.732816 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:24.733173 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:24.734038 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:24.734044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:24.734064 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:24.734075 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:24.734130 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:24.734151 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:24.734308 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:24.734382 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 18:58:24.734555 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":166,"threads_started":1}
I20260812 18:58:24.736052 9311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:24.736937 9311 log.cc:824] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:24.738370 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:24.739166 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150: No bootstrap required, opened a new log
I20260812 18:58:24.742483 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:24.740690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.60:43971 (local address 127.0.0.1:38327)
0812 18:58:24.740820 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:24.740826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:24.741035 (+ 209us) server_negotiation.cc:407] Connection header received
0812 18:58:24.741156 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:24.741159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:24.741199 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:24.741292 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:24.742070 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:24.742075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:24.742078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:24.742092 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:24.742123 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:24.742136 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:24.742224 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:24.742299 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 18:58:24.742345 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 18:58:24.743741 9311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } }
I20260812 18:58:24.743984 9311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:24.744019 9311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0faf8043c4184425a7669c4146379150, State: Initialized, Role: FOLLOWER
I20260812 18:58:24.744410 9311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } }
I20260812 18:58:24.744531 9311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:24.744649 9311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:24.744771 9311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:24.745806 9311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } }
I20260812 18:58:24.746181 9311 leader_election.cc:302] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0faf8043c4184425a7669c4146379150; no voters:
I20260812 18:58:24.746551 9311 leader_election.cc:288] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:24.746690 9325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:24.746945 9325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [term 1 LEADER]: Becoming Leader. State: Replica: 0faf8043c4184425a7669c4146379150, State: Running, Role: LEADER
I20260812 18:58:24.747295 9325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } }
I20260812 18:58:24.747946 9311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:24.749130 9327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0faf8043c4184425a7669c4146379150. Latest consensus state: current_term: 1 leader_uuid: "0faf8043c4184425a7669c4146379150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } } }
I20260812 18:58:24.749266 9327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:24.749586 9326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0faf8043c4184425a7669c4146379150" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0faf8043c4184425a7669c4146379150" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38327 } } }
I20260812 18:58:24.749750 9326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:24.749737 9337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:24.753888 9337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:24.760774 9337 catalog_manager.cc:1347] Generated new cluster ID: 9b687ac9f87c42dfb212721158a3a0e6
I20260812 18:58:24.760877 9337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:24.789135 9337 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:24.789625 9337 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:24.796216 9337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0faf8043c4184425a7669c4146379150: Generated new TSK 0
I20260812 18:58:24.797233 9337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:24.809525 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9229
2026-08-12T18:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1560 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:24 dist-test-slave-rnrw krb5kdc[9350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:24 dist-test-slave-rnrw krb5kdc[9350](info): set up 2 sockets
Aug 12 18:58:24 dist-test-slave-rnrw krb5kdc[9350](info): commencing operation
krb5kdc: starting...
W20260812 18:58:25.945613 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:26.088877 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40709
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:26.207870 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:26.208184 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:26.208228 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:26.212441 9366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:26.212527 9366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:26.212574 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:26.212595 9366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:26.212611 9366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:26.218428 9366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:40709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:26.219691 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:26.220857 9366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:26.230253 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:26.230257 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:26.230257 9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:26.230612 9366 server_base.cc:1034] running on GCE node
I20260812 18:58:26.231292 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:26.232604 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:26.233821 9366 hybrid_clock.cc:648] HybridClock initialized: now 1786561106233789 us; error 46 us; skew 500 ppm
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561106, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:26.245678 9366 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:26.246954 9366 webserver.cc:460] Webserver started at http://127.16.40.254:40815/ using document root <none> and password file <none>
I20260812 18:58:26.247629 9366 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:26.247699 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:26.247889 9366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:26.250240 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "5cb5d616199c442fb2f9d28ff8f249e3"
format_stamp: "Formatted at 2026-08-12 18:58:26 on dist-test-slave-rnrw"
server_key: "9d1b981ec8f888e24c188f01259a1023"
server_key_iv: "d146a4d5a27c72df2f98ca3e5abb76df"
server_key_version: "encryptionkey@0"
I20260812 18:58:26.250809 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "5cb5d616199c442fb2f9d28ff8f249e3"
format_stamp: "Formatted at 2026-08-12 18:58:26 on dist-test-slave-rnrw"
server_key: "9d1b981ec8f888e24c188f01259a1023"
server_key_iv: "d146a4d5a27c72df2f98ca3e5abb76df"
server_key_version: "encryptionkey@0"
I20260812 18:58:26.255132 9366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:26.258260 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:26.259745 9366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 18:58:26.259925 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "5cb5d616199c442fb2f9d28ff8f249e3"
format_stamp: "Formatted at 2026-08-12 18:58:26 on dist-test-slave-rnrw"
server_key: "9d1b981ec8f888e24c188f01259a1023"
server_key_iv: "d146a4d5a27c72df2f98ca3e5abb76df"
server_key_version: "encryptionkey@0"
I20260812 18:58:26.260066 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:26.286259 9366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:26.290021 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:26.290248 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:26.297925 9366 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:40709
I20260812 18:58:26.297959 9444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:40709 every 8 connection(s)
I20260812 18:58:26.298954 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:26.301985 9445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:26.307837 9445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: Bootstrap starting.
I20260812 18:58:26.308292 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9366
I20260812 18:58:26.308435 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:26.308771 16547 external_mini_cluster.cc:1428] Setting key b731b234e2d2a2c86632a52b0fb03a09
I20260812 18:58:26.310783 9445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:26.311705 9445 log.cc:824] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:26.314073 9445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: No bootstrap required, opened a new log
I20260812 18:58:26.318114 9445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } }
I20260812 18:58:26.318377 9445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:26.318410 9445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb5d616199c442fb2f9d28ff8f249e3, State: Initialized, Role: FOLLOWER
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:26.318939 9445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } }
I20260812 18:58:26.319132 9445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:26.319198 9445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:26.319312 9445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:26.320690 9445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } }
I20260812 18:58:26.321100 9445 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb5d616199c442fb2f9d28ff8f249e3; no voters:
I20260812 18:58:26.321482 9445 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:26.321628 9450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:26.321867 9450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb5d616199c442fb2f9d28ff8f249e3, State: Running, Role: LEADER
I20260812 18:58:26.322243 9450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } }
I20260812 18:58:26.322963 9445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:26.324256 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } } }
I20260812 18:58:26.324380 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:26.324733 9452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb5d616199c442fb2f9d28ff8f249e3. Latest consensus state: current_term: 1 leader_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5d616199c442fb2f9d28ff8f249e3" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40709 } } }
I20260812 18:58:26.324833 9452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:26.326546 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:26.310270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47269 (local address 127.16.40.254:40709)
0812 18:58:26.310775 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:26.310797 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:26.310844 (+ 47us) server_negotiation.cc:407] Connection header received
0812 18:58:26.311737 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:26.311762 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:26.312217 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:26.312561 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:26.313355 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:26.314800 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:26.315707 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:26.316407 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:26.316609 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:26.319459 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:26.319466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:26.319484 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:26.319521 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:26.321731 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:26.322419 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:26.322425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:26.322432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:26.322579 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:26.323029 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:26.323035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:26.323039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:26.323589 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:26.324911 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:26.325111 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 18:58:26.325378 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":140,"threads_started":1}
W20260812 18:58:26.328634 9467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:26.328771 9467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:26.328860 9459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:26.329586 9459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:26.329622 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:26.336344 9459 catalog_manager.cc:1347] Generated new cluster ID: 73c7ccc002114b66ad245ed7aeafe641
I20260812 18:58:26.336436 9459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:26.345865 9448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:26.331737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54083 (local address 127.16.40.254:40709)
0812 18:58:26.331854 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:26.331859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:26.332160 (+ 301us) server_negotiation.cc:407] Connection header received
0812 18:58:26.332290 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:26.332293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:26.332343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:26.332429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:26.333341 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:26.334443 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:26.335343 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:26.335597 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:26.335718 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:26.341920 (+ 6202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:26.341925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:26.341928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:26.341978 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:26.343778 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:26.344381 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:26.344385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:26.344387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:26.344486 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:26.344885 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:26.344888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:26.344890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:26.345136 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:26.345286 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:26.345574 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 18:58:26.345687 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 18:58:26.349794 9459 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:26.351162 9459 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:26.359294 9459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb5d616199c442fb2f9d28ff8f249e3: Generated new TSK 0
I20260812 18:58:26.360267 9459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:26.368072 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9366
W20260812 18:58:26.378795 9348 connection.cc:535] client connection to 127.16.40.254:40709 recv error: Network error: failed to read from TLS socket (remote: 127.16.40.254:40709): Connection reset by peer (error 104)
2026-08-12T18:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9479](info): set up 2 sockets
Aug 12 18:58:26 dist-test-slave-rnrw krb5kdc[9479](info): commencing operation
krb5kdc: starting...
W20260812 18:58:27.530239 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:27 dist-test-slave-rnrw krb5kdc[9479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:27.675172 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38025
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:35271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:38025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:27.793743 9496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:27.794091 9496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:27.794180 9496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:27.798763 9496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:27.798840 9496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:27.798861 9496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:27.798878 9496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:27.798893 9496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:27.804638 9496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:35271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:38025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:27.805896 9496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:27.807019 9496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:27.816562 9505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:27.816546 9502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:27.816607 9503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:27.816776 9496 server_base.cc:1034] running on GCE node
I20260812 18:58:27.817534 9496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:27.818689 9496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:27.819872 9496 hybrid_clock.cc:648] HybridClock initialized: now 1786561107819855 us; error 34 us; skew 500 ppm
Aug 12 18:58:27 dist-test-slave-rnrw krb5kdc[9479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561107, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:27.822940 9496 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:27.823911 9496 webserver.cc:460] Webserver started at http://127.16.40.254:42371/ using document root <none> and password file <none>
I20260812 18:58:27.824474 9496 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:27.824591 9496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:27.824806 9496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:27.826969 9496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "63c97299ca7240d6981731139fbedec2"
format_stamp: "Formatted at 2026-08-12 18:58:27 on dist-test-slave-rnrw"
server_key: "3f4e9485a2e314f9e70058961ab37bf1"
server_key_iv: "5659588d7a03083b7ff33ff56dbd4f6a"
server_key_version: "encryptionkey@0"
I20260812 18:58:27.827580 9496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "63c97299ca7240d6981731139fbedec2"
format_stamp: "Formatted at 2026-08-12 18:58:27 on dist-test-slave-rnrw"
server_key: "3f4e9485a2e314f9e70058961ab37bf1"
server_key_iv: "5659588d7a03083b7ff33ff56dbd4f6a"
server_key_version: "encryptionkey@0"
I20260812 18:58:27.831976 9496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 18:58:27.834739 9515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:27.836162 9496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 18:58:27.836315 9496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "63c97299ca7240d6981731139fbedec2"
format_stamp: "Formatted at 2026-08-12 18:58:27 on dist-test-slave-rnrw"
server_key: "3f4e9485a2e314f9e70058961ab37bf1"
server_key_iv: "5659588d7a03083b7ff33ff56dbd4f6a"
server_key_version: "encryptionkey@0"
I20260812 18:58:27.836442 9496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:27.870743 9496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:27.874318 9496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:27.874550 9496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:27.882654 9496 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:38025
I20260812 18:58:27.882664 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:38025 every 8 connection(s)
I20260812 18:58:27.883713 9496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:27.886847 9579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:27.892900 9579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: Bootstrap starting.
I20260812 18:58:27.893622 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9496
I20260812 18:58:27.893774 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:27.894063 16547 external_mini_cluster.cc:1428] Setting key 1564beaf88c93ed3cd2a72bc309951db
I20260812 18:58:27.896075 9579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:27.896971 9579 log.cc:824] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:27.899601 9579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: No bootstrap required, opened a new log
I20260812 18:58:27.904346 9579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } }
I20260812 18:58:27.904703 9579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:27.904767 9579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c97299ca7240d6981731139fbedec2, State: Initialized, Role: FOLLOWER
I20260812 18:58:27.905139 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:27.895539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36297 (local address 127.16.40.254:38025)
0812 18:58:27.896007 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:27.896023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:27.896061 (+ 38us) server_negotiation.cc:407] Connection header received
0812 18:58:27.897007 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:27.897030 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:27.897462 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:27.897769 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:27.898262 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:27.899706 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:27.900743 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:27.901459 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:27.901627 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:27.902581 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:27.902592 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:27.902617 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:27.903234 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:27.903313 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:27.903345 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:27.903490 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:27.903812 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 18:58:27.903975 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":151,"threads_started":1}
I20260812 18:58:27.905279 9579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } }
I20260812 18:58:27.905457 9579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:27.905551 9579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:27.905678 9579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:27.906679 9579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } }
I20260812 18:58:27.907052 9579 leader_election.cc:302] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c97299ca7240d6981731139fbedec2; no voters:
I20260812 18:58:27.907467 9579 leader_election.cc:288] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:27.907565 9585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:27.907819 9585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [term 1 LEADER]: Becoming Leader. State: Replica: 63c97299ca7240d6981731139fbedec2, State: Running, Role: LEADER
I20260812 18:58:27.908231 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } }
I20260812 18:58:27.908593 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:27.908949 9579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:27.911839 9587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c97299ca7240d6981731139fbedec2. Latest consensus state: current_term: 1 leader_uuid: "63c97299ca7240d6981731139fbedec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } } }
I20260812 18:58:27.912006 9587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:27.912308 9586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c97299ca7240d6981731139fbedec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c97299ca7240d6981731139fbedec2" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 38025 } } }
I20260812 18:58:27.912402 9586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2 [sys.catalog]: This master's current role is: LEADER
W20260812 18:58:27.915213 9608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:27.915314 9608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:27.915385 9609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:27.916072 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:27.911082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40361 (local address 127.16.40.254:38025)
0812 18:58:27.911215 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:27.911221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:27.911624 (+ 403us) server_negotiation.cc:407] Connection header received
0812 18:58:27.911940 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:27.911943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:27.912025 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:27.912102 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:27.912732 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:27.913274 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:27.914136 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:27.914387 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:27.914507 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:27.915561 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:27.915565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:27.915568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:27.915584 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:27.915619 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:27.915630 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:27.915732 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:27.915864 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 18:58:27.915903 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 18:58:27.916388 9609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:27.923000 9609 catalog_manager.cc:1347] Generated new cluster ID: 15135ff1da84468a8316634f6c9dd136
I20260812 18:58:27.923094 9609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:27.936177 9609 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:27.937606 9609 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:27.955811 9609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63c97299ca7240d6981731139fbedec2: Generated new TSK 0
I20260812 18:58:27.956856 9609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:27.961565 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9496
W20260812 18:58:27.972318 9477 connection.cc:535] client connection to 127.16.40.254:38025 recv error: Network error: failed to read from TLS socket (remote: 127.16.40.254:38025): Connection reset by peer (error 104)
2026-08-12T18:58:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1593 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:28 dist-test-slave-rnrw krb5kdc[9615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:28 dist-test-slave-rnrw krb5kdc[9615](info): set up 2 sockets
Aug 12 18:58:28 dist-test-slave-rnrw krb5kdc[9615](info): commencing operation
krb5kdc: starting...
W20260812 18:58:29.110268 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:29 dist-test-slave-rnrw krb5kdc[9615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:29.249960 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40085
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:29.368422 9630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:29.368803 9630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:29.368894 9630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:29.373034 9630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:29.373157 9630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:29.373198 9630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:29.373236 9630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:29.373274 9630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:29.378793 9630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:40085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:29.379974 9630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:29.381083 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:29.389823 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:29.389858 9639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:29.389861 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:29.390147 9630 server_base.cc:1034] running on GCE node
I20260812 18:58:29.390712 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:29.391875 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:29.393071 9630 hybrid_clock.cc:648] HybridClock initialized: now 1786561109393042 us; error 43 us; skew 500 ppm
Aug 12 18:58:29 dist-test-slave-rnrw krb5kdc[9615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:29.396106 9630 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:29.397229 9630 webserver.cc:460] Webserver started at http://127.16.40.254:38627/ using document root <none> and password file <none>
I20260812 18:58:29.397851 9630 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:29.397951 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:29.398205 9630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:29.400379 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "0ea0e620030e46e9b43c618ecfe76707"
format_stamp: "Formatted at 2026-08-12 18:58:29 on dist-test-slave-rnrw"
server_key: "acbcb6c851bf7eeef9c2a611eb779801"
server_key_iv: "f423e473bf9ef8400185a6bb039df108"
server_key_version: "encryptionkey@0"
I20260812 18:58:29.401036 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "0ea0e620030e46e9b43c618ecfe76707"
format_stamp: "Formatted at 2026-08-12 18:58:29 on dist-test-slave-rnrw"
server_key: "acbcb6c851bf7eeef9c2a611eb779801"
server_key_iv: "f423e473bf9ef8400185a6bb039df108"
server_key_version: "encryptionkey@0"
I20260812 18:58:29.405040 9630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:29.407655 9647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:29.409088 9630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:29.409253 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "0ea0e620030e46e9b43c618ecfe76707"
format_stamp: "Formatted at 2026-08-12 18:58:29 on dist-test-slave-rnrw"
server_key: "acbcb6c851bf7eeef9c2a611eb779801"
server_key_iv: "f423e473bf9ef8400185a6bb039df108"
server_key_version: "encryptionkey@0"
I20260812 18:58:29.409386 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:29.414443 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:29.418087 9630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:29.418319 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:29.426672 9630 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:40085
I20260812 18:58:29.426688 9712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:40085 every 8 connection(s)
I20260812 18:58:29.427781 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:29.431355 9713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:29.438038 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9630
I20260812 18:58:29.438234 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:29.438578 16547 external_mini_cluster.cc:1428] Setting key 86969ce27b9554c4d3e88c3bc15db22b
I20260812 18:58:29.439031 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707: Bootstrap starting.
I20260812 18:58:29.442256 9713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:29.443347 9713 log.cc:824] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:29.444988 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:29.440368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40485 (local address 127.16.40.254:40085)
0812 18:58:29.440993 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:29.441009 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:29.441051 (+ 42us) server_negotiation.cc:407] Connection header received
0812 18:58:29.441791 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:29.441810 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:29.442158 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:29.442559 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:29.443536 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:29.443543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:29.443561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:29.443583 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:29.443647 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:29.443669 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:29.443807 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:29.443917 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 18:58:29.444110 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":231,"threads_started":1}
I20260812 18:58:29.446169 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707: No bootstrap required, opened a new log
I20260812 18:58:29.448465 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:29.451026 9713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } }
I20260812 18:58:29.451320 9713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:29.451403 9713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea0e620030e46e9b43c618ecfe76707, State: Initialized, Role: FOLLOWER
I20260812 18:58:29.451944 9713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } }
I20260812 18:58:29.452157 9713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:29.452245 9713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:29.452414 9713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:29.453122 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:29.450816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37911 (local address 127.16.40.254:40085)
0812 18:58:29.450975 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:29.450981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:29.451230 (+ 249us) server_negotiation.cc:407] Connection header received
0812 18:58:29.451434 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:29.451437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:29.451490 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:29.451545 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:29.452592 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:29.452596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:29.452601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:29.452619 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:29.452655 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:29.452667 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:29.452778 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:29.452922 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 18:58:29.452986 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 18:58:29.453923 9713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } }
I20260812 18:58:29.454348 9713 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea0e620030e46e9b43c618ecfe76707; no voters:
I20260812 18:58:29.454773 9713 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:29.454901 9723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:29.455174 9723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea0e620030e46e9b43c618ecfe76707, State: Running, Role: LEADER
I20260812 18:58:29.455566 9723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } }
I20260812 18:58:29.456305 9713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:29.457494 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea0e620030e46e9b43c618ecfe76707. Latest consensus state: current_term: 1 leader_uuid: "0ea0e620030e46e9b43c618ecfe76707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } } }
I20260812 18:58:29.457672 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:29.457559 9724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea0e620030e46e9b43c618ecfe76707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea0e620030e46e9b43c618ecfe76707" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 40085 } } }
I20260812 18:58:29.458369 9724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:29.458977 9733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:29.462402 9733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:29.468997 9733 catalog_manager.cc:1347] Generated new cluster ID: 92a5cf37af534ea1ad398a7b60c32973
I20260812 18:58:29.469106 9733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:29.481938 9733 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:29.482373 9733 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:29.490340 9733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea0e620030e46e9b43c618ecfe76707: Generated new TSK 0
I20260812 18:58:29.491335 9733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:29.504217 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9630
2026-08-12T18:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:29 dist-test-slave-rnrw krb5kdc[9745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:29 dist-test-slave-rnrw krb5kdc[9745](info): set up 2 sockets
Aug 12 18:58:29 dist-test-slave-rnrw krb5kdc[9745](info): commencing operation
krb5kdc: starting...
W20260812 18:58:30.673208 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:30 dist-test-slave-rnrw krb5kdc[9745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:30.820844 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33879
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:30.940563 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:30.940929 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:30.941016 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:30.945293 9760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:30.945401 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:30.945441 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:30.945500 9760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:30.945544 9760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:30.951272 9760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:30.952527 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:30.953708 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:30.962893 9766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:30.963074 9760 server_base.cc:1034] running on GCE node
W20260812 18:58:30.962896 9767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:30.962896 9771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:30.963903 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:30.965212 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:30.966445 9760 hybrid_clock.cc:648] HybridClock initialized: now 1786561110966432 us; error 29 us; skew 500 ppm
Aug 12 18:58:30 dist-test-slave-rnrw krb5kdc[9745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:30.969667 9760 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:30.970708 9760 webserver.cc:460] Webserver started at http://127.16.40.254:46473/ using document root <none> and password file <none>
I20260812 18:58:30.971336 9760 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:30.971432 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:30.971696 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:30.974020 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "1aab7bf6b7774c71964294556f8ec514"
format_stamp: "Formatted at 2026-08-12 18:58:30 on dist-test-slave-rnrw"
server_key: "03d8c1a71c62600741b744ceb445189b"
server_key_iv: "0a8c6c3e8a698d23972c3725fe017bdd"
server_key_version: "encryptionkey@0"
I20260812 18:58:30.974638 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "1aab7bf6b7774c71964294556f8ec514"
format_stamp: "Formatted at 2026-08-12 18:58:30 on dist-test-slave-rnrw"
server_key: "03d8c1a71c62600741b744ceb445189b"
server_key_iv: "0a8c6c3e8a698d23972c3725fe017bdd"
server_key_version: "encryptionkey@0"
I20260812 18:58:30.978890 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 18:58:30.981698 9778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:30.983255 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:30.983446 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "1aab7bf6b7774c71964294556f8ec514"
format_stamp: "Formatted at 2026-08-12 18:58:30 on dist-test-slave-rnrw"
server_key: "03d8c1a71c62600741b744ceb445189b"
server_key_iv: "0a8c6c3e8a698d23972c3725fe017bdd"
server_key_version: "encryptionkey@0"
I20260812 18:58:30.983595 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:30.995383 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:30.999065 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:30.999298 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:31.007093 9760 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:33879
I20260812 18:58:31.007167 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:33879 every 8 connection(s)
I20260812 18:58:31.008229 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:31.008941 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9760
I20260812 18:58:31.009096 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:31.009415 16547 external_mini_cluster.cc:1428] Setting key 29f2eb8d36484a2d6b9d6ee49e6f32b1
I20260812 18:58:31.011678 9842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:31.018823 9842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514: Bootstrap starting.
Aug 12 18:58:31 dist-test-slave-rnrw krb5kdc[9745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:31.021806 9842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:31.022832 9842 log.cc:824] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:31.025520 9842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514: No bootstrap required, opened a new log
I20260812 18:58:31.026436 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:31.011040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59897 (local address 127.16.40.254:33879)
0812 18:58:31.011512 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:31.011536 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:31.011821 (+ 285us) server_negotiation.cc:407] Connection header received
0812 18:58:31.012811 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:31.012837 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:31.013289 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:31.013686 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:31.014424 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:31.015967 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:31.017095 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:31.017614 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:31.017803 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:31.020802 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:31.020809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:31.020827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:31.020864 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:31.023116 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:31.023675 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:31.023681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:31.023686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:31.023829 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:31.024111 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:31.024114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:31.024116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:31.024676 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:31.024884 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:31.025097 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 18:58:31.025432 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":164,"threads_started":1}
I20260812 18:58:31.029768 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:31.030285 9842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } }
I20260812 18:58:31.030633 9842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:31.030735 9842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aab7bf6b7774c71964294556f8ec514, State: Initialized, Role: FOLLOWER
I20260812 18:58:31.031289 9842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } }
I20260812 18:58:31.031546 9842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:31.031628 9842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:31.031737 9842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:31.033221 9842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } }
I20260812 18:58:31.033633 9842 leader_election.cc:302] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aab7bf6b7774c71964294556f8ec514; no voters:
I20260812 18:58:31.034005 9842 leader_election.cc:288] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:31.034128 9852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:31.034396 9852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [term 1 LEADER]: Becoming Leader. State: Replica: 1aab7bf6b7774c71964294556f8ec514, State: Running, Role: LEADER
I20260812 18:58:31.034780 9852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } }
I20260812 18:58:31.035661 9842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:31.037019 9854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aab7bf6b7774c71964294556f8ec514. Latest consensus state: current_term: 1 leader_uuid: "1aab7bf6b7774c71964294556f8ec514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } } }
I20260812 18:58:31.037202 9854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:31.037016 9853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aab7bf6b7774c71964294556f8ec514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aab7bf6b7774c71964294556f8ec514" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 33879 } } }
I20260812 18:58:31.037814 9853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:31.038108 9866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:31.042373 9866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:31.043380 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:31.032285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45391 (local address 127.16.40.254:33879)
0812 18:58:31.032409 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:31.032414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:31.032611 (+ 197us) server_negotiation.cc:407] Connection header received
0812 18:58:31.032809 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:31.032815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:31.032877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:31.032988 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:31.033912 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:31.034720 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:31.036182 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:31.036603 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:31.036735 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:31.038774 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:31.038778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:31.038781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:31.038814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:31.041202 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:31.041853 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:31.041857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:31.041859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:31.041966 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:31.042329 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:31.042332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:31.042334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:31.042598 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:31.042740 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:31.043039 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 18:58:31.043209 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 18:58:31.049759 9866 catalog_manager.cc:1347] Generated new cluster ID: 0170e6718ef948dca91d0e06b24fa9d7
I20260812 18:58:31.049861 9866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:31.064160 9866 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:31.065681 9866 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:31.074018 9866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aab7bf6b7774c71964294556f8ec514: Generated new TSK 0
I20260812 18:58:31.075009 9866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:31.090809 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9760
2026-08-12T18:58:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:31 dist-test-slave-rnrw krb5kdc[9876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:31 dist-test-slave-rnrw krb5kdc[9876](info): set up 2 sockets
Aug 12 18:58:31 dist-test-slave-rnrw krb5kdc[9876](info): commencing operation
krb5kdc: starting...
W20260812 18:58:32.227043 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:32 dist-test-slave-rnrw krb5kdc[9876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:32.368814 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45213
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:32.488265 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:32.488633 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:32.488682 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:32.492884 9889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:32.492975 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:32.492995 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:32.493012 9889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:32.493028 9889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:32.498747 9889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:32.499959 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:32.501070 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 18:58:32.510740 9889 server_base.cc:1034] running on GCE node
W20260812 18:58:32.510790 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:32.510998 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:32.510740 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:32.511786 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:32.513104 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:32.514298 9889 hybrid_clock.cc:648] HybridClock initialized: now 1786561112514284 us; error 32 us; skew 500 ppm
Aug 12 18:58:32 dist-test-slave-rnrw krb5kdc[9876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561112, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:32.517602 9889 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:32.518725 9889 webserver.cc:460] Webserver started at http://127.16.40.254:42217/ using document root <none> and password file <none>
I20260812 18:58:32.519366 9889 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:32.519436 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:32.519639 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:32.522023 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "b733b5469a44403ca57d343cc0d6a915"
format_stamp: "Formatted at 2026-08-12 18:58:32 on dist-test-slave-rnrw"
server_key: "6db0ea7c200c4c1a6d225bfa7d25065b"
server_key_iv: "80dec14ea6693c4e6dfaae048d0143ad"
server_key_version: "encryptionkey@0"
I20260812 18:58:32.522686 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "b733b5469a44403ca57d343cc0d6a915"
format_stamp: "Formatted at 2026-08-12 18:58:32 on dist-test-slave-rnrw"
server_key: "6db0ea7c200c4c1a6d225bfa7d25065b"
server_key_iv: "80dec14ea6693c4e6dfaae048d0143ad"
server_key_version: "encryptionkey@0"
I20260812 18:58:32.527599 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 18:58:32.530759 9906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:32.532627 9889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 18:58:32.532837 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "b733b5469a44403ca57d343cc0d6a915"
format_stamp: "Formatted at 2026-08-12 18:58:32 on dist-test-slave-rnrw"
server_key: "6db0ea7c200c4c1a6d225bfa7d25065b"
server_key_iv: "80dec14ea6693c4e6dfaae048d0143ad"
server_key_version: "encryptionkey@0"
I20260812 18:58:32.532989 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:32.547049 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:32.550901 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:32.551149 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:32.559387 9889 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:45213
I20260812 18:58:32.559419 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:45213 every 8 connection(s)
I20260812 18:58:32.560565 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:32.563737 9970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:32.567225 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 9889
I20260812 18:58:32.567368 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:32.567732 16547 external_mini_cluster.cc:1428] Setting key 479ac0560a266630470871d0570f2c71
I20260812 18:58:32.570784 9970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915: Bootstrap starting.
I20260812 18:58:32.573832 9970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:32.574981 9970 log.cc:824] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:32.578027 9970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915: No bootstrap required, opened a new log
I20260812 18:58:32.579367 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:32.569378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56885 (local address 127.16.40.254:45213)
0812 18:58:32.569879 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:32.569901 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:32.569949 (+ 48us) server_negotiation.cc:407] Connection header received
0812 18:58:32.570820 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:32.570846 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:32.571331 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:32.571725 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:32.572210 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:32.573748 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:32.575235 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:32.576036 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:32.576220 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:32.577553 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:32.577559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:32.577574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:32.577969 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:32.578027 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:32.578050 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:32.578184 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:32.578350 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 18:58:32.578586 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":131,"threads_started":1}
I20260812 18:58:32.582079 9970 raft_consensus.cc:348] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } }
I20260812 18:58:32.582368 9970 raft_consensus.cc:374] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:32.582463 9970 raft_consensus.cc:729] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b733b5469a44403ca57d343cc0d6a915, State: Initialized, Role: FOLLOWER
I20260812 18:58:32.582435 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:32.582901 9970 consensus_queue.cc:260] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } }
I20260812 18:58:32.583024 9970 raft_consensus.cc:388] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:32.583112 9970 raft_consensus.cc:482] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:32.583256 9970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:32.584357 9970 raft_consensus.cc:504] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } }
I20260812 18:58:32.584812 9970 leader_election.cc:302] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b733b5469a44403ca57d343cc0d6a915; no voters:
I20260812 18:58:32.585239 9970 leader_election.cc:288] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:32.585345 9981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:32.585656 9981 raft_consensus.cc:686] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [term 1 LEADER]: Becoming Leader. State: Replica: b733b5469a44403ca57d343cc0d6a915, State: Running, Role: LEADER
I20260812 18:58:32.586041 9981 consensus_queue.cc:237] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } }
I20260812 18:58:32.586694 9970 sys_catalog.cc:564] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:32.587708 9982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b733b5469a44403ca57d343cc0d6a915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } } }
I20260812 18:58:32.587857 9982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:32.588227 9983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b733b5469a44403ca57d343cc0d6a915. Latest consensus state: current_term: 1 leader_uuid: "b733b5469a44403ca57d343cc0d6a915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b733b5469a44403ca57d343cc0d6a915" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 45213 } } }
I20260812 18:58:32.588318 9983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:32.588835 9991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:32.589717 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:32.584771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40611 (local address 127.16.40.254:45213)
0812 18:58:32.584927 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:32.584932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:32.585167 (+ 235us) server_negotiation.cc:407] Connection header received
0812 18:58:32.585284 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:32.585287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:32.585341 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:32.585449 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:32.585787 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:32.586519 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:32.587753 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:32.588167 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:32.588271 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:32.589231 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:32.589236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:32.589239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:32.589254 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:32.589282 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:32.589298 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:32.589352 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:32.589518 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 18:58:32.589573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":55}
I20260812 18:58:32.592137 9991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:32.597827 9991 catalog_manager.cc:1347] Generated new cluster ID: f37216a6fa6f4ad4bae64b57d6d4d9e4
I20260812 18:58:32.597903 9991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:32.627250 9991 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:32.628361 9991 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:32.633572 9991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b733b5469a44403ca57d343cc0d6a915: Generated new TSK 0
I20260812 18:58:32.634323 9991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:32.643469 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 9889
2026-08-12T18:58:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:32 dist-test-slave-rnrw krb5kdc[10006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:32 dist-test-slave-rnrw krb5kdc[10006](info): set up 2 sockets
Aug 12 18:58:32 dist-test-slave-rnrw krb5kdc[10006](info): commencing operation
krb5kdc: starting...
W20260812 18:58:33.764739 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:33 dist-test-slave-rnrw krb5kdc[10006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:33.899988 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:34975
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:34975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:34.015414 10020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:34.015714 10020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:34.015758 10020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:34.019873 10020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:34.019944 10020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:34.019964 10020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:34.019987 10020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:34.020002 10020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:34.025473 10020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:34975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:34.026667 10020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:34.027779 10020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:34.036916 10031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:34.036933 10028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:34.036929 10020 server_base.cc:1034] running on GCE node
W20260812 18:58:34.036947 10027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:34.037848 10020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:34.039038 10020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:34.040251 10020 hybrid_clock.cc:648] HybridClock initialized: now 1786561114040220 us; error 44 us; skew 500 ppm
Aug 12 18:58:34 dist-test-slave-rnrw krb5kdc[10006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:34.043231 10020 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:34.044274 10020 webserver.cc:460] Webserver started at http://127.16.40.254:42151/ using document root <none> and password file <none>
I20260812 18:58:34.044950 10020 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:34.045047 10020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:34.045303 10020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:34.047458 10020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "dd1a6068fd04459e85156967e4841fc4"
format_stamp: "Formatted at 2026-08-12 18:58:34 on dist-test-slave-rnrw"
server_key: "22fd17ea173143bcabb7b0d5ef8f43f6"
server_key_iv: "0763a9a1696d4dc1e499a88130e2f49e"
server_key_version: "encryptionkey@0"
I20260812 18:58:34.048064 10020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "dd1a6068fd04459e85156967e4841fc4"
format_stamp: "Formatted at 2026-08-12 18:58:34 on dist-test-slave-rnrw"
server_key: "22fd17ea173143bcabb7b0d5ef8f43f6"
server_key_iv: "0763a9a1696d4dc1e499a88130e2f49e"
server_key_version: "encryptionkey@0"
I20260812 18:58:34.052104 10020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 18:58:34.054853 10038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:34.056116 10020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 18:58:34.056269 10020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "dd1a6068fd04459e85156967e4841fc4"
format_stamp: "Formatted at 2026-08-12 18:58:34 on dist-test-slave-rnrw"
server_key: "22fd17ea173143bcabb7b0d5ef8f43f6"
server_key_iv: "0763a9a1696d4dc1e499a88130e2f49e"
server_key_version: "encryptionkey@0"
I20260812 18:58:34.056394 10020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:34.061273 10020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:34.064419 10020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:34.064667 10020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:34.072042 10020 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:34975
I20260812 18:58:34.072124 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:34975 every 8 connection(s)
I20260812 18:58:34.073136 10020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:34.076180 10100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:34.077790 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10020
I20260812 18:58:34.077972 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:34.078287 16547 external_mini_cluster.cc:1428] Setting key 08d73dc03d1b6996819d9affc5a569dc
I20260812 18:58:34.084539 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4: Bootstrap starting.
I20260812 18:58:34.084789 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:34.080024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43951 (local address 127.16.40.254:34975)
0812 18:58:34.080576 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:34.080600 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:34.080650 (+ 50us) server_negotiation.cc:407] Connection header received
0812 18:58:34.081598 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:34.081626 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:34.082039 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:34.082452 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:34.083190 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:34.083196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:34.083213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:34.083226 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:34.083294 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:34.083323 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:34.083491 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:34.083568 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 18:58:34.083732 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":122,"threads_started":1}
I20260812 18:58:34.087180 10100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:34.087689 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:34.088301 10100 log.cc:824] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:34.091192 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4: No bootstrap required, opened a new log
I20260812 18:58:34.092355 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:34.090305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48691 (local address 127.16.40.254:34975)
0812 18:58:34.090440 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:34.090449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:34.090691 (+ 242us) server_negotiation.cc:407] Connection header received
0812 18:58:34.090846 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:34.090848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:34.090882 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:34.090977 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:34.091880 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:34.091883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:34.091885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 18:58:34.091896 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:34.091924 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 18:58:34.091936 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:34.092032 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:34.092167 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 18:58:34.092206 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:58:34.095983 10100 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } }
I20260812 18:58:34.096293 10100 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 18:58:34.096364 10100 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1a6068fd04459e85156967e4841fc4, State: Initialized, Role: FOLLOWER
I20260812 18:58:34.096920 10100 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } }
I20260812 18:58:34.097110 10100 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 18:58:34.097188 10100 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 18:58:34.097301 10100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:34.098506 10100 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } }
I20260812 18:58:34.098958 10100 leader_election.cc:302] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1a6068fd04459e85156967e4841fc4; no voters:
I20260812 18:58:34.099434 10100 leader_election.cc:288] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 18:58:34.099675 10112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:34.100067 10112 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [term 1 LEADER]: Becoming Leader. State: Replica: dd1a6068fd04459e85156967e4841fc4, State: Running, Role: LEADER
I20260812 18:58:34.100423 10112 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } }
I20260812 18:58:34.101109 10100 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:34.102504 10113 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1a6068fd04459e85156967e4841fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } } }
I20260812 18:58:34.102656 10113 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:34.103049 10114 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1a6068fd04459e85156967e4841fc4. Latest consensus state: current_term: 1 leader_uuid: "dd1a6068fd04459e85156967e4841fc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1a6068fd04459e85156967e4841fc4" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 34975 } } }
I20260812 18:58:34.103211 10114 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:34.103133 10121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:34.106417 10121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:34.112596 10121 catalog_manager.cc:1347] Generated new cluster ID: 7198e95310c046bda6e52e9174da9743
I20260812 18:58:34.112687 10121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:34.128293 10121 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:34.128727 10121 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:34.138868 10121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd1a6068fd04459e85156967e4841fc4: Generated new TSK 0
I20260812 18:58:34.139629 10121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 18:58:34.145291 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10020
2026-08-12T18:58:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1502 ms)
[----------] 18 tests from AuthTokenIssuingTest (28172 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:34 dist-test-slave-rnrw krb5kdc[10138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:34 dist-test-slave-rnrw krb5kdc[10138](info): set up 2 sockets
Aug 12 18:58:34 dist-test-slave-rnrw krb5kdc[10138](info): commencing operation
krb5kdc: starting...
W20260812 18:58:35.274026 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:35.412992 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:42539
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:35.541582 10153 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:35.541945 10153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:35.541970 10153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:35.542017 10153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:35.546365 10153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:35.546577 10153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:35.546624 10153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:35.546644 10153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:35.546661 10153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:35.552640 10153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:35.553844 10153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:35.554929 10153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:35.564582 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:35.564582 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:35.564610 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:35.565078 10153 server_base.cc:1034] running on GCE node
I20260812 18:58:35.565701 10153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:35.566999 10153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:35.568249 10153 hybrid_clock.cc:648] HybridClock initialized: now 1786561115568214 us; error 52 us; skew 500 ppm
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:35.571754 10153 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:35.573016 10153 webserver.cc:460] Webserver started at http://127.16.40.254:42899/ using document root <none> and password file <none>
I20260812 18:58:35.573673 10153 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:35.573776 10153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:35.574035 10153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:35.576333 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "c8987725842842dfadf84f84f5480171"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "1d7149c79bd89987d98572a2fef272d9"
server_key_iv: "05bf185d69509a4d0f0ab1808f161fcf"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.577047 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "c8987725842842dfadf84f84f5480171"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "1d7149c79bd89987d98572a2fef272d9"
server_key_iv: "05bf185d69509a4d0f0ab1808f161fcf"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.581372 10153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:58:35.584226 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:35.585767 10153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:35.585965 10153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "c8987725842842dfadf84f84f5480171"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "1d7149c79bd89987d98572a2fef272d9"
server_key_iv: "05bf185d69509a4d0f0ab1808f161fcf"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.586117 10153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:35.598214 10153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:35.601979 10153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:35.602203 10153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:35.610246 10153 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:42539
I20260812 18:58:35.610543 10240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:42539 every 8 connection(s)
I20260812 18:58:35.611429 10153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:35.611886 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10153
I20260812 18:58:35.612035 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:35.612325 16547 external_mini_cluster.cc:1428] Setting key 375b63edb1f2b3adf3af5888d4d858f3
I20260812 18:58:35.614902 10241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:35.619235 10241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.253@KRBTEST.COM" created.
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:35.637579 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.622982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.16.40.254:42539)
0812 18:58:35.623286 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:35.623297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:35.623341 (+ 44us) server_negotiation.cc:407] Connection header received
0812 18:58:35.624266 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:35.624272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:35.624514 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:35.624763 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:35.625637 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.626637 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.628370 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.629417 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.629690 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.632282 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:35.632289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:35.632301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:35.632322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:35.634155 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.634744 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.634750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.634752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.634869 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.635204 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.635209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.635211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.635642 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:35.635845 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:35.636225 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 18:58:35.636646 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":203,"spinlock_wait_cycles":6912,"thread_start_us":86,"threads_started":1}
I20260812 18:58:35.637584 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.622049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42539 (local address 127.0.0.1:43770)
0812 18:58:35.622698 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.622770 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:35.623902 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:35.624764 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:35.624769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:35.625221 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:35.625482 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.625495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.626793 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.626798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.628076 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.628090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.629601 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.629604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.629695 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.630457 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:35.630475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:35.632103 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:35.634340 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.634351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.634365 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.634620 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.634976 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.634979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.634981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.635106 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.635818 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:35.635825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:35.635973 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 18:58:35.636143 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 18:58:35.636700 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":280,"spinlock_wait_cycles":7936,"thread_start_us":164,"threads_started":1}
I20260812 18:58:35.640424 10241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:35.641477 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.641135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41858)
0812 18:58:35.641280 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.641398 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 18:58:35.641733 10173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
I20260812 18:58:35.642719 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.642486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41874)
0812 18:58:35.642601 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.642655 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 18:58:35.642887 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:44365: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
I20260812 18:58:35.644668 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:44365
--webserver_interface=127.16.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:58:35.692147 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } attempt: 1
I20260812 18:58:35.693164 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.692793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41882)
0812 18:58:35.692974 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.693064 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 18:58:35.693987 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.693739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41884)
0812 18:58:35.693856 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.693917 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 18:58:35.694193 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:44365: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
I20260812 18:58:35.762405 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } attempt: 2
I20260812 18:58:35.763381 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.763019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41892)
0812 18:58:35.763208 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.763282 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 18:58:35.764312 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.764050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41894)
0812 18:58:35.764180 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.764241 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 18:58:35.764515 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:44365: Network error: Client connection negotiation failed: client connection to 127.16.40.253:44365: connect: Connection refused (error 111)
W20260812 18:58:35.768380 10248 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:35.768834 10248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:35.768887 10248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:35.768929 10248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:35.773341 10248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:35.773411 10248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:35.773430 10248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:35.773447 10248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:35.773463 10248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:35.781749 10248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:35.783456 10248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:35.785167 10248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:35.797369 10259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:35.797353 10257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:35.797366 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:35.797657 10248 server_base.cc:1034] running on GCE node
I20260812 18:58:35.798429 10248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:35.800055 10248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:35.801263 10248 hybrid_clock.cc:648] HybridClock initialized: now 1786561115801227 us; error 58 us; skew 500 ppm
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:35.804423 10248 init.cc:377] Logged in from keytab as kudu/127.16.40.253@KRBTEST.COM (short username kudu)
I20260812 18:58:35.805490 10248 webserver.cc:460] Webserver started at http://127.16.40.253:33935/ using document root <none> and password file <none>
I20260812 18:58:35.806211 10248 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:35.806294 10248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:35.806502 10248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:35.809116 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data/instance:
uuid: "f68ce81f6662446c87a41aad809c2705"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "80cea16579fdd9e898ae34c9393461b0"
server_key_iv: "d0906b2cf2ee571c6b20c127290278c0"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.809805 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal/instance:
uuid: "f68ce81f6662446c87a41aad809c2705"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "80cea16579fdd9e898ae34c9393461b0"
server_key_iv: "d0906b2cf2ee571c6b20c127290278c0"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.813769 10248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:35.816334 10267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:35.817687 10248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 18:58:35.817826 10248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "f68ce81f6662446c87a41aad809c2705"
format_stamp: "Formatted at 2026-08-12 18:58:35 on dist-test-slave-rnrw"
server_key: "80cea16579fdd9e898ae34c9393461b0"
server_key_iv: "d0906b2cf2ee571c6b20c127290278c0"
server_key_version: "encryptionkey@0"
I20260812 18:58:35.817935 10248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:35.831059 10248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:35.834375 10248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:35.834601 10248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:35.842442 10248 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.253:44365
I20260812 18:58:35.842746 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.253:44365 every 8 connection(s)
I20260812 18:58:35.843506 10248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
I20260812 18:58:35.846673 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:35.850842 10327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:35.853547 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10248
I20260812 18:58:35.853675 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-1/wal/instance
I20260812 18:58:35.853982 16547 external_mini_cluster.cc:1428] Setting key aae48b4f53d7f3c2b2841ee3131e4b9a
I20260812 18:58:35.855751 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } attempt: 3
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.40.252@KRBTEST.COM; defaulting to no policy
I20260812 18:58:35.867357 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.856233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41906)
0812 18:58:35.856431 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.856447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:35.856537 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:35.857257 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:35.857263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:35.857479 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:35.857635 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.857646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.858803 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.858809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.859334 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.859368 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.860201 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.860205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.860302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.860880 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:35.860904 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:35.862646 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:35.865287 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.865294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.865490 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.865835 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.865841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.865926 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.866709 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:35.866713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:35.866899 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 18:58:35.867020 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 18:58:35.867169 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 18:58:35.867786 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.852902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.16.40.254:42539)
0812 18:58:35.853059 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:35.853063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:35.854149 (+ 1086us) server_negotiation.cc:407] Connection header received
0812 18:58:35.855149 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:35.855153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:35.855217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:35.855326 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:35.857106 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.857781 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.859667 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.859994 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.860117 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.863373 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:35.863376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:35.863378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:35.863400 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:35.865122 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865634 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.865641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.865643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.865729 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.866069 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.866072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.866074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.866332 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:35.866447 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:35.867507 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0812 18:58:35.867639 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.252@KRBTEST.COM" created.
I20260812 18:58:35.868353 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.853308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42539 (local address 127.0.0.1:43784)
0812 18:58:35.853963 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.854009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:35.854916 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:35.855515 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:35.855523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:35.856101 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:35.856898 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.856921 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.857913 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.857920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.859519 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.859535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.860180 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.860185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.860411 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.861095 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:35.861124 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:35.863225 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:35.865240 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.865260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.865497 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.865837 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.865847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.865960 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.866447 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:35.866455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:35.866615 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 18:58:35.866807 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 18:58:35.867210 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":184,"thread_start_us":205,"threads_started":1}
I20260812 18:58:35.868351 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.856318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.16.40.253:44365)
0812 18:58:35.856809 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:35.856819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:35.856838 (+ 19us) server_negotiation.cc:407] Connection header received
0812 18:58:35.856872 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:35.856875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:35.857060 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:35.857242 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:35.857798 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.858680 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.859523 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.860055 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.860557 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.862811 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:35.862818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:35.862835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:35.862862 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:35.865196 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.865606 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.865612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.865614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.865734 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.866024 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.866028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.866030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.866588 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:35.866735 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:35.867107 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 18:58:35.867270 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":168,"threads_started":1}
I20260812 18:58:35.870675 10241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:35.871475 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.871201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57378)
0812 18:58:35.871328 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.871384 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 18:58:35.871579 10327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:35.872541 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.872231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57390)
0812 18:58:35.872426 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.872478 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260812 18:58:35.872754 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Aug 12 18:58:35 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:35.881448 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.872325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.16.40.253:44365)
0812 18:58:35.872426 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:35.872432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:35.872446 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:58:35.872482 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:35.872485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:35.872528 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:35.872652 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:35.873267 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.873838 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.875096 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.875347 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.875449 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.877915 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:35.877918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:35.877920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:35.877945 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:35.879539 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.880001 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.880004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.880005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.880101 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:35.880322 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:35.880324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:35.880325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:35.880756 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:35.880883 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:35.881091 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 18:58:35.881254 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":37}
I20260812 18:58:35.881449 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.872093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41918)
0812 18:58:35.872201 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.872215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:35.872301 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:35.872746 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:35.872752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:35.872966 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:35.873121 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.873132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.873962 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.873969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.874927 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:35.874942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:35.875490 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:35.875496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:35.875632 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:35.876165 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:35.876186 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:35.877751 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:35.879654 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.879657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.879658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.879920 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.880183 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:35.880186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:35.880187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:35.880263 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:35.880874 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:35.880876 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:35.880963 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 18:58:35.881050 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 18:58:35.881178 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 18:58:35.882436 10327 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:35.883344 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.883025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57406)
0812 18:58:35.883174 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.883269 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 18:58:35.884093 10271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
I20260812 18:58:35.885192 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.884943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57422)
0812 18:58:35.885080 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.885131 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 18:58:35.885483 10327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
I20260812 18:58:35.888300 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:43101
--webserver_interface=127.16.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:58:35.903936 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 1
I20260812 18:58:35.904995 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.904679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57430)
0812 18:58:35.904836 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.904909 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 18:58:35.905857 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.905637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57442)
0812 18:58:35.905750 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.905797 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 18:58:35.906033 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
I20260812 18:58:35.934808 10327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 1
I20260812 18:58:35.935860 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.935523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57450)
0812 18:58:35.935709 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.935774 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 18:58:35.936694 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.936437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57460)
0812 18:58:35.936584 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.936640 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 18:58:35.936861 10327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
I20260812 18:58:35.981245 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 2
I20260812 18:58:35.982249 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.981906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57470)
0812 18:58:35.982098 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.982164 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 18:58:35.983131 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:35.982855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57476)
0812 18:58:35.983017 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:35.983068 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 18:58:35.983350 10241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
I20260812 18:58:36.005048 10327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 2
I20260812 18:58:36.006148 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.005756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57492)
0812 18:58:36.005993 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.006055 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260812 18:58:36.007040 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.006779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57500)
0812 18:58:36.006910 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.006970 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 18:58:36.007187 10327 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:43101: Network error: Client connection negotiation failed: client connection to 127.16.40.252:43101: connect: Connection refused (error 111)
W20260812 18:58:36.011225 10336 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:36.011559 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:36.011585 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:36.011673 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:36.016132 10336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:36.016204 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:36.016224 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:36.016242 10336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:36.016258 10336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:36.022053 10336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:42539,127.16.40.253:44365,127.16.40.252:43101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:43101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:36.023288 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:36.024421 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:36.033679 10343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:36.033694 10342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:36.033692 10345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:36.034006 10336 server_base.cc:1034] running on GCE node
I20260812 18:58:36.034657 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:36.035867 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:36.037041 10336 hybrid_clock.cc:648] HybridClock initialized: now 1786561116037030 us; error 29 us; skew 500 ppm
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561116, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:36.040107 10336 init.cc:377] Logged in from keytab as kudu/127.16.40.252@KRBTEST.COM (short username kudu)
I20260812 18:58:36.041221 10336 webserver.cc:460] Webserver started at http://127.16.40.252:40587/ using document root <none> and password file <none>
I20260812 18:58:36.041872 10336 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:36.041970 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:36.042220 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:36.044442 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data/instance:
uuid: "b713a502bf694027b2d5772e1d715c0c"
format_stamp: "Formatted at 2026-08-12 18:58:36 on dist-test-slave-rnrw"
server_key: "5e1529ce3929ecb403348736004bd539"
server_key_iv: "9c42147f2de3cfce768caeb6e8994c22"
server_key_version: "encryptionkey@0"
I20260812 18:58:36.045167 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal/instance:
uuid: "b713a502bf694027b2d5772e1d715c0c"
format_stamp: "Formatted at 2026-08-12 18:58:36 on dist-test-slave-rnrw"
server_key: "5e1529ce3929ecb403348736004bd539"
server_key_iv: "9c42147f2de3cfce768caeb6e8994c22"
server_key_version: "encryptionkey@0"
I20260812 18:58:36.049712 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 18:58:36.053325 10353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:36.055105 10336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 18:58:36.055349 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "b713a502bf694027b2d5772e1d715c0c"
format_stamp: "Formatted at 2026-08-12 18:58:36 on dist-test-slave-rnrw"
server_key: "5e1529ce3929ecb403348736004bd539"
server_key_iv: "9c42147f2de3cfce768caeb6e8994c22"
server_key_version: "encryptionkey@0"
I20260812 18:58:36.055513 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:36.068531 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:36.072072 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:36.072309 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:36.080298 10336 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.252:43101
I20260812 18:58:36.080729 10412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.252:43101 every 8 connection(s)
I20260812 18:58:36.081452 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
I20260812 18:58:36.082602 10241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 3
I20260812 18:58:36.085042 10413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:36.086756 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10336
I20260812 18:58:36.086879 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786560811542710-16547-0/minicluster-data/master-2/wal/instance
I20260812 18:58:36.087167 16547 external_mini_cluster.cc:1428] Setting key 743f03e41303c69e291ead1c2a61ff13
I20260812 18:58:36.090387 10413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:58:36.097198 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.083154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57504)
0812 18:58:36.083303 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.083322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:36.083425 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:36.085971 (+ 2546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:36.085974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:36.086216 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:36.086340 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.086348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.087951 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.087955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.088797 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.088811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.089842 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.089846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.089943 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.090528 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:36.090545 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:36.092285 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:36.094771 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.094778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.094781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.095063 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.095533 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.095537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.095540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.095653 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.096604 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:36.096607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:36.096720 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 18:58:36.096848 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 18:58:36.097021 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 18:58:36.098415 10327 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } attempt: 3
I20260812 18:58:36.098496 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.083997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.16.40.252:43101)
0812 18:58:36.084477 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.084493 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.084530 (+ 37us) server_negotiation.cc:407] Connection header received
0812 18:58:36.085331 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.085350 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.085689 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.086027 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.086508 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.087777 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.088937 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.089692 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.089868 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.092512 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.092520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.092562 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.092601 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.094628 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.095222 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.095231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.095238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.095396 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.095744 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.095751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.095754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.096412 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.096647 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.097021 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.097360 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":331,"thread_start_us":166,"threads_started":1}
I20260812 18:58:36.099269 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.087910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.16.40.252:43101)
0812 18:58:36.088300 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.088311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.088429 (+ 118us) server_negotiation.cc:407] Connection header received
0812 18:58:36.088582 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.088585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.088631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.088748 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.089576 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.090278 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.091625 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.091960 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.092070 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.095169 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.095172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.095175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.095201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.097216 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.097747 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.097751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.097753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.097839 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.098167 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.098170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.098172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.098376 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.098559 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.098924 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.099049 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561116, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:36.104241 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: Bootstrap starting.
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:58:36.107580 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.092878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.16.40.254:42539)
0812 18:58:36.093252 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.093257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.094926 (+ 1669us) server_negotiation.cc:407] Connection header received
0812 18:58:36.095105 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.095111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.095170 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.095251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.096158 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.096999 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.098545 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.098859 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.098939 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.103275 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.103279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.103282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.103307 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.105122 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.105828 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.105833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.105836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.105957 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.106287 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.106293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.106297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.106609 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.106710 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.107233 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.107372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260812 18:58:36.107800 10241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:36.108261 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.092841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:42539 (local address 127.0.0.1:43798)
0812 18:58:36.094807 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.094850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:36.094992 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:36.095297 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:36.095305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:36.095790 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:36.095989 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.095999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.097170 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.097176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.098405 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.098417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.099546 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.099549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.099946 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.100968 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:36.100982 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:36.103097 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:36.105253 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.105265 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.105271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.105694 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.106082 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.106084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.106086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.106191 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.106775 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:36.106783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:36.106938 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 18:58:36.107745 (+ 807us) client_negotiation.cc:240] Negotiation successful
0812 18:58:36.107973 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260812 18:58:36.108915 10241 log.cc:824] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:36.109691 10413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:36.111088 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.099586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57518)
0812 18:58:36.099730 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.099747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:36.099833 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:36.100402 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:36.100406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:36.100639 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:36.100750 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.100757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.102047 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.102053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.102669 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.102683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.103331 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.103335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.103456 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.104134 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:36.104150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:36.105763 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:36.108184 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.108188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.108190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.108465 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.109977 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.109981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.109983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.110101 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.110628 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:36.110632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:36.110754 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 18:58:36.110838 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 18:58:36.110966 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 18:58:36.111682 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.100054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.16.40.252:43101)
0812 18:58:36.100159 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.100168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.100184 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:58:36.100226 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.100229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.100269 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.100355 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.100882 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.101892 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.102835 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.103183 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.103703 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.105908 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.105912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.105921 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.105959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.107613 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.109742 (+ 2129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.109746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.109748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.109846 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.110250 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.110254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.110255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.110499 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.111390 (+ 891us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.111473 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.111572 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 18:58:36.111904 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: No bootstrap required, opened a new log
I20260812 18:58:36.114465 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: Bootstrap starting.
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561116, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:36.116912 10241 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.117197 10241 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8987725842842dfadf84f84f5480171, State: Initialized, Role: FOLLOWER
I20260812 18:58:36.117550 10327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:36.117590 10241 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.118613 10327 log.cc:824] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:36.119257 10241 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:36.120635 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.110754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.16.40.253:44365)
0812 18:58:36.110869 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.110876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.110892 (+ 16us) server_negotiation.cc:407] Connection header received
0812 18:58:36.110932 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.110936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.110989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.111065 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.111633 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.112353 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.113448 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.113779 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.113877 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.116590 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.116594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.116597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.116633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.118347 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.119088 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.119093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.119098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.119190 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.119534 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.119540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.119544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.119814 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.119894 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.120291 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.120413 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 18:58:36.120688 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.102422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.16.40.254:42539)
0812 18:58:36.102790 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.102807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.102821 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:58:36.102856 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.102859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.102903 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.104600 (+ 1697us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.104739 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.105591 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.107291 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.107653 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.115341 (+ 7688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.115493 (+ 152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.115497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.115500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.115518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.117939 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.118563 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.118566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.118568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.118665 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.119006 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.119010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.119011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.119255 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.119444 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.120307 (+ 863us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.120460 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":109,"threads_started":1}
I20260812 18:58:36.121166 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.110272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:44365 (local address 127.0.0.1:41920)
0812 18:58:36.110524 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.110538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:36.110612 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:36.111186 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:36.111189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:36.111366 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:36.111494 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.111502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.112490 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.112494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.113304 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.113316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.114024 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.114027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.114129 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.114697 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:36.114712 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:36.116392 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:36.118641 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.118645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.118647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.118910 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.119301 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.119305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.119307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.119409 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.119983 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:36.119988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:36.120090 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 18:58:36.120805 (+ 715us) client_negotiation.cc:240] Negotiation successful
0812 18:58:36.120985 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260812 18:58:36.121363 10422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.121513 10422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.122211 10413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:36.123036 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: No bootstrap required, opened a new log
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561116, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
Aug 12 18:58:36 dist-test-slave-rnrw krb5kdc[10138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:36.129386 10327 raft_consensus.cc:348] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.129957 10327 raft_consensus.cc:729] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68ce81f6662446c87a41aad809c2705, State: Initialized, Role: FOLLOWER
I20260812 18:58:36.126506 10153 master_runner.cc:186] Error getting master registration for 127.16.40.253:44365: OK, instance_id { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" instance_seqno: 1786561115840190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:58:36.130501 10153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 18:58:36.130822 10327 consensus_queue.cc:260] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
W20260812 18:58:36.132230 10437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:36.132367 10437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:36.132823 10327 sys_catalog.cc:564] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:36.133111 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.122724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:43101 (local address 127.0.0.1:57534)
0812 18:58:36.122894 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:36.122906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:36.122987 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:36.123565 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:36.123568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:36.123744 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:36.123842 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.123849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.124928 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.124931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.125708 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:36.125717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.126291 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.126294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:36.126368 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.126912 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:36.126926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:36.129212 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:36.131526 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.131530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.131532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.131771 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.132034 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:36.132036 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:36.132038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:36.132113 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:36.132599 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:36.132602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:36.132691 (+ 89us) client_negotiation.cc:769] Sending connection context
0812 18:58:36.132788 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 18:58:36.132945 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 18:58:36.133620 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.122753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.16.40.252:43101)
0812 18:58:36.122869 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.122873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.122947 (+ 74us) server_negotiation.cc:407] Connection header received
0812 18:58:36.123081 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.123084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.123212 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.123992 (+ 780us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.124085 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.124825 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.125845 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.126177 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.126614 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.129345 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.129349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.129351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.129377 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.131379 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.131870 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.131873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.131875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.131948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.132201 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.132204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.132206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.132436 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.133272 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.133348 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.133464 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 18:58:36.136899 10413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: Bootstrap starting.
I20260812 18:58:36.137207 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.137432 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 18:58:36.138517 10455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:36.138603 10455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:36.139122 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.122744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.16.40.253:44365)
0812 18:58:36.122847 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.122851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.122864 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:58:36.122897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.122899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.122939 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.123018 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.123871 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.124697 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.125849 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.126177 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.126276 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.129977 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.129982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.129985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.130021 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.131743 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.132283 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.132286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.132288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.132389 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.132782 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.132787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.132789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.133011 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.133195 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.138795 (+ 5600us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.138961 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 18:58:36.139874 10413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:36.140859 10413 log.cc:824] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:36.142967 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:36.140818 10248 master_runner.cc:186] Error getting master registration for 127.16.40.252:43101: OK, instance_id { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" instance_seqno: 1786561116078107 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:58:36.144946 10248 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 18:58:36.144917 10211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8987725842842dfadf84f84f5480171"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810
I20260812 18:58:36.145282 10211 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 18:58:36.145417 10211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:36.146905 10211 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.148893 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8987725842842dfadf84f84f5480171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f68ce81f6662446c87a41aad809c2705"
I20260812 18:58:36.149008 10211 leader_election.cc:288] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [CANDIDATE]: Term 1 election: Requested vote from peers f68ce81f6662446c87a41aad809c2705 (127.16.40.253:44365), b713a502bf694027b2d5772e1d715c0c (127.16.40.252:43101)
I20260812 18:58:36.149207 10299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:36.150794 10299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8987725842842dfadf84f84f5480171 in term 1.
I20260812 18:58:36.151396 10173 leader_election.cc:302] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8987725842842dfadf84f84f5480171, f68ce81f6662446c87a41aad809c2705; no voters:
I20260812 18:58:36.151780 10422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:36.152016 10422 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [term 1 LEADER]: Becoming Leader. State: Replica: c8987725842842dfadf84f84f5480171, State: Running, Role: LEADER
I20260812 18:58:36.152427 10422 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.154580 10422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8987725842842dfadf84f84f5480171. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.154706 10422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:36.155218 10470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:36.157193 10470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:36.158905 10413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: No bootstrap required, opened a new log
I20260812 18:58:36.163789 10413 raft_consensus.cc:348] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.164126 10413 raft_consensus.cc:729] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b713a502bf694027b2d5772e1d715c0c, State: Initialized, Role: FOLLOWER
I20260812 18:58:36.164512 10413 consensus_queue.cc:260] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } }
I20260812 18:58:36.165285 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.165624 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.166213 10413 sys_catalog.cc:564] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:36.166273 10384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8987725842842dfadf84f84f5480171" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b713a502bf694027b2d5772e1d715c0c"
I20260812 18:58:36.166514 10384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:36.168067 10384 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8987725842842dfadf84f84f5480171 in term 1.
I20260812 18:58:36.178359 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.153906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.16.40.252:43101)
0812 18:58:36.154027 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.154033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.168236 (+ 14203us) server_negotiation.cc:407] Connection header received
0812 18:58:36.168581 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.168585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.168637 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.168692 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.170037 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.170820 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.172060 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.172400 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.172489 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.174463 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.174480 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.174483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.174513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.176315 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.176900 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.176904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.176906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.176999 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.177300 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.177304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.177306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.177534 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.177668 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.178039 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.178167 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 18:58:36.178960 10484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:36.179117 10484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:36.182629 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.153022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.16.40.254:42539)
0812 18:58:36.153135 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.153140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.167969 (+ 14829us) server_negotiation.cc:407] Connection header received
0812 18:58:36.168834 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.168837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.168890 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.168939 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.171958 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.172723 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.175804 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.176184 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.176269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.178641 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.178645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.178648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.178681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.180585 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.181134 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.181138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.181140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.181236 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.181573 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.181576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.181578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.181816 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.181932 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.182296 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.182443 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 18:58:36.185581 10384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [term 1 FOLLOWER]: Refusing update from remote peer c8987725842842dfadf84f84f5480171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:36.186523 10299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [term 1 FOLLOWER]: Refusing update from remote peer c8987725842842dfadf84f84f5480171: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:36.187530 10422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:36.188011 10422 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:36.189874 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:36.151885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.16.40.253:44365)
0812 18:58:36.152011 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.152015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.159570 (+ 7555us) server_negotiation.cc:407] Connection header received
0812 18:58:36.160589 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.160594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.160660 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.160719 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.165609 (+ 4890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.166424 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.170949 (+ 4525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:36.171298 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:36.171386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:36.185694 (+ 14308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:36.185698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:36.185701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:36.185735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:36.187569 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.188086 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.188090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.188092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.188181 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:36.188487 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:36.188490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:36.188493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:36.188776 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:36.188912 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:36.189585 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 18:58:36.189708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:58:36.193084 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8987725842842dfadf84f84f5480171. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.193220 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.197953 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8987725842842dfadf84f84f5480171. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.198096 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.203126 10465 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.203271 10465 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:36.204236 10422 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.204356 10422 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:36.206046 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.206183 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.207291 10439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8987725842842dfadf84f84f5480171" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8987725842842dfadf84f84f5480171" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 42539 } } peers { permanent_uuid: "f68ce81f6662446c87a41aad809c2705" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 44365 } } peers { permanent_uuid: "b713a502bf694027b2d5772e1d715c0c" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 43101 } } }
I20260812 18:58:36.207444 10439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:36.207417 10491 mvcc.cc:205] Tried to move back new op lower bound from 7317754331883393024 to 7317754331764727808. Current Snapshot: MvccSnapshot[applied={T|T < 7317754331883393024}]
I20260812 18:58:36.207818 10493 mvcc.cc:205] Tried to move back new op lower bound from 7317754331883393024 to 7317754331764727808. Current Snapshot: MvccSnapshot[applied={T|T < 7317754331883393024}]
I20260812 18:58:36.208007 10471 mvcc.cc:205] Tried to move back new op lower bound from 7317754331883393024 to 7317754331764727808. Current Snapshot: MvccSnapshot[applied={T|T < 7317754331883393024}]
I20260812 18:58:36.211803 10470 catalog_manager.cc:1347] Generated new cluster ID: 0ba8bd6e3569489e82dc484318e466eb
I20260812 18:58:36.211877 10470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:36.246915 10470 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:36.248632 10470 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:36.266963 10470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8987725842842dfadf84f84f5480171: Generated new TSK 0
I20260812 18:58:36.268169 10470 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:58:36.307570 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.304128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.16.40.253:44365)
0812 18:58:36.304267 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.304272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.306381 (+ 2109us) server_negotiation.cc:407] Connection header received
0812 18:58:36.306601 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.306605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.306667 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.306720 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.307421 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41956: BlockingRecv error: recv got EOF from 127.0.0.1:41956 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 18:58:36.309423 10420 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.305047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.16.40.254:42539)
0812 18:58:36.305170 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.305174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.308000 (+ 2826us) server_negotiation.cc:407] Connection header received
0812 18:58:36.308411 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.308415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.308470 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.308524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.309298 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43824: BlockingRecv error: recv got EOF from 127.0.0.1:43824 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 18:58:36.310787 10415 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.305719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.16.40.252:43101)
0812 18:58:36.305839 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.305844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.308323 (+ 2479us) server_negotiation.cc:407] Connection header received
0812 18:58:36.310002 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.310006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.310058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.310109 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.310655 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57550: BlockingRecv error: recv got EOF from 127.0.0.1:57550 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 18:58:36.316242 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.314295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.16.40.253:44365)
0812 18:58:36.314455 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.314463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.315023 (+ 560us) server_negotiation.cc:407] Connection header received
0812 18:58:36.315267 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.315270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.315352 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.315404 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.316127 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41968: BlockingRecv error: recv got EOF from 127.0.0.1:41968 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 18:58:36.321869 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.319996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.16.40.253:44365)
0812 18:58:36.320134 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.320139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.320756 (+ 617us) server_negotiation.cc:407] Connection header received
0812 18:58:36.320969 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.320973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.321029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.321083 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.321729 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41972: BlockingRecv error: recv got EOF from 127.0.0.1:41972 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 18:58:36.328367 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.326595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.16.40.253:44365)
0812 18:58:36.326712 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.326717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.327312 (+ 595us) server_negotiation.cc:407] Connection header received
0812 18:58:36.327545 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.327549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.327605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.327658 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.328234 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 18:58:36.336623 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.334799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.16.40.253:44365)
0812 18:58:36.334951 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.334956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.335595 (+ 639us) server_negotiation.cc:407] Connection header received
0812 18:58:36.335813 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.335817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.335872 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.335923 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.336471 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 18:58:36.349458 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.347333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.16.40.253:44365)
0812 18:58:36.347517 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.347522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.348389 (+ 867us) server_negotiation.cc:407] Connection header received
0812 18:58:36.348641 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.348645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.348702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.348756 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.349335 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42008: BlockingRecv error: recv got EOF from 127.0.0.1:42008 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 18:58:36.370030 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.368561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.16.40.253:44365)
0812 18:58:36.368719 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.368724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.369033 (+ 309us) server_negotiation.cc:407] Connection header received
0812 18:58:36.369190 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.369194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.369268 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.369383 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.369887 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42020: BlockingRecv error: recv got EOF from 127.0.0.1:42020 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 18:58:36.405478 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.404196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.16.40.253:44365)
0812 18:58:36.404361 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.404366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.404594 (+ 228us) server_negotiation.cc:407] Connection header received
0812 18:58:36.404740 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.404743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.404794 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.404868 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.405348 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42036: BlockingRecv error: recv got EOF from 127.0.0.1:42036 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 18:58:36.473064 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.471834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.16.40.253:44365)
0812 18:58:36.472004 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.472009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.472117 (+ 108us) server_negotiation.cc:407] Connection header received
0812 18:58:36.472244 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.472247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.472293 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.472364 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.472915 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42040: BlockingRecv error: recv got EOF from 127.0.0.1:42040 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 18:58:36.604297 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.603159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.16.40.253:44365)
0812 18:58:36.603320 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.603324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.603376 (+ 52us) server_negotiation.cc:407] Connection header received
0812 18:58:36.603515 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.603517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.603567 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.603642 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.604176 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42042: BlockingRecv error: recv got EOF from 127.0.0.1:42042 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 18:58:36.863943 10332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:36.862646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.16.40.253:44365)
0812 18:58:36.862802 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:36.862816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:36.862857 (+ 41us) server_negotiation.cc:407] Connection header received
0812 18:58:36.863025 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:36.863028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:36.863077 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:36.863149 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:36.863812 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42052: BlockingRecv error: recv got EOF from 127.0.0.1:42052 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 18:58:37.140094 10455 catalog_manager.cc:1259] Loaded cluster ID: 0ba8bd6e3569489e82dc484318e466eb
I20260812 18:58:37.140192 10455 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: loading cluster ID for follower catalog manager: success
I20260812 18:58:37.142063 10455 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: acquiring CA information for follower catalog manager: success
I20260812 18:58:37.143322 10455 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f68ce81f6662446c87a41aad809c2705: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:58:37.180732 10484 catalog_manager.cc:1259] Loaded cluster ID: 0ba8bd6e3569489e82dc484318e466eb
I20260812 18:58:37.180827 10484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: loading cluster ID for follower catalog manager: success
I20260812 18:58:37.182794 10484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: acquiring CA information for follower catalog manager: success
I20260812 18:58:37.184091 10484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b713a502bf694027b2d5772e1d715c0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:58:37.382169 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:37.378249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.16.40.253:44365)
0812 18:58:37.378514 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:37.378518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:37.378579 (+ 61us) server_negotiation.cc:407] Connection header received
0812 18:58:37.378771 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:37.378774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:37.378926 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:37.379051 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:37.379340 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.379882 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.380765 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.380992 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.381080 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:37.381360 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:37.381815 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:37.381907 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:37.381969 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 18:58:37.382021 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260812 18:58:37.389117 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:37.385403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.16.40.252:43101)
0812 18:58:37.385604 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:37.385636 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:37.385649 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:58:37.385791 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:37.385794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:37.385920 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:37.386026 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:37.386250 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.386790 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.387699 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.387921 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.388003 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:37.388298 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:37.388776 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:37.388870 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:37.388933 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 18:58:37.388984 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":56,"threads_started":1}
I20260812 18:58:37.397256 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:37.392487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.16.40.254:42539)
0812 18:58:37.392781 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:37.392785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:37.393188 (+ 403us) server_negotiation.cc:407] Connection header received
0812 18:58:37.393429 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:37.393432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:37.393625 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:37.393728 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:37.394121 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.394756 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.395762 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.395999 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.396109 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:37.396457 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:37.396818 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:37.396923 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:37.397003 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 18:58:37.397066 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260812 18:58:37.397478 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:37.393103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.16.40.252:43101)
0812 18:58:37.393538 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:37.393542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:37.393556 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:58:37.393719 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:37.393723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:37.393776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:37.393827 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:37.394190 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.394925 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.396180 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.396518 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.396618 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:37.396969 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:37.397049 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:37.397097 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:37.397275 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 18:58:37.397327 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260812 18:58:37.397625 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:37.392495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.16.40.253:44365)
0812 18:58:37.392922 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:37.392927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:37.393305 (+ 378us) server_negotiation.cc:407] Connection header received
0812 18:58:37.393373 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:37.393376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:37.393433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:37.393508 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:37.393876 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.394724 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.396102 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:37.396438 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:37.396513 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:37.396978 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:37.397049 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:37.397097 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:37.397418 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 18:58:37.397497 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260812 18:58:37.406138 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10153
I20260812 18:58:37.418107 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10248
I20260812 18:58:37.428064 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10336
2026-08-12T18:58:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3284 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 18:58:37 dist-test-slave-rnrw krb5kdc[10598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 18:58:37 dist-test-slave-rnrw krb5kdc[10598](info): set up 2 sockets
Aug 12 18:58:37 dist-test-slave-rnrw krb5kdc[10598](info): commencing operation
krb5kdc: starting...
W20260812 18:58:38.555398 16547 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/test-user.keytab.
Aug 12 18:58:38 dist-test-slave-rnrw krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T18:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T18:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.40.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
Entry for principal kudu/127.16.40.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab.
I20260812 18:58:38.693665 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41595
--webserver_interface=127.16.40.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 18:58:38.814149 10613 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:38.814512 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:38.814543 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:38.814638 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:38.818925 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:38.819010 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:38.819031 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:38.819048 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:38.819064 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:38.824754 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.254.keytab
--principal=kudu/127.16.40.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.254:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.40.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:38.825953 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:38.827032 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:38.836997 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:38.837060 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:38.836997 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:38.837451 10613 server_base.cc:1034] running on GCE node
I20260812 18:58:38.838109 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:38.839460 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:38.840647 10613 hybrid_clock.cc:648] HybridClock initialized: now 1786561118840609 us; error 52 us; skew 500 ppm
Aug 12 18:58:38 dist-test-slave-rnrw krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:38.843837 10613 init.cc:377] Logged in from keytab as kudu/127.16.40.254@KRBTEST.COM (short username kudu)
I20260812 18:58:38.844942 10613 webserver.cc:460] Webserver started at http://127.16.40.254:38557/ using document root <none> and password file <none>
I20260812 18:58:38.845575 10613 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:38.845674 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:38.845930 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:38.848265 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data/instance:
uuid: "6d9f4f7d0dc7404b961802b3a3808533"
format_stamp: "Formatted at 2026-08-12 18:58:38 on dist-test-slave-rnrw"
server_key: "b7a01f634e0c96087984c9d1335cbae9"
server_key_iv: "7ffddfbf727a5b2569cc78cc6ac0f8e7"
server_key_version: "encryptionkey@0"
I20260812 18:58:38.848937 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance:
uuid: "6d9f4f7d0dc7404b961802b3a3808533"
format_stamp: "Formatted at 2026-08-12 18:58:38 on dist-test-slave-rnrw"
server_key: "b7a01f634e0c96087984c9d1335cbae9"
server_key_iv: "7ffddfbf727a5b2569cc78cc6ac0f8e7"
server_key_version: "encryptionkey@0"
I20260812 18:58:38.853188 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 18:58:38.856123 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:38.857473 10613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 18:58:38.857642 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal
uuid: "6d9f4f7d0dc7404b961802b3a3808533"
format_stamp: "Formatted at 2026-08-12 18:58:38 on dist-test-slave-rnrw"
server_key: "b7a01f634e0c96087984c9d1335cbae9"
server_key_iv: "7ffddfbf727a5b2569cc78cc6ac0f8e7"
server_key_version: "encryptionkey@0"
I20260812 18:58:38.857775 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:38.873152 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:38.876751 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:38.876994 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:38.884512 10613 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.254:41595
I20260812 18:58:38.884850 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.254:41595 every 8 connection(s)
I20260812 18:58:38.885699 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/data/info.pb
I20260812 18:58:38.888770 10694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:38.891878 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10613
I20260812 18:58:38.892045 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-0/wal/instance
I20260812 18:58:38.892354 16547 external_mini_cluster.cc:1428] Setting key 9d8a35496426bc2253aee3fb197690c3
I20260812 18:58:38.892824 10694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.40.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.253@KRBTEST.COM" created.
Aug 12 18:58:38 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:38.911448 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.895315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.16.40.254:41595)
0812 18:58:38.895913 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:38.895927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:38.896038 (+ 111us) server_negotiation.cc:407] Connection header received
0812 18:58:38.897194 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:38.897203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:38.897483 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:38.897717 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:38.898515 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:38.900071 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:38.901226 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:38.901876 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:38.902082 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:38.906101 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:38.906109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:38.906122 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:38.906147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:38.908015 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:38.908576 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:38.908583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:38.908585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:38.908711 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:38.909046 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:38.909049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:38.909051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:38.909471 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:38.909633 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:38.910017 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 18:58:38.910348 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":464,"thread_start_us":174,"threads_started":1}
I20260812 18:58:38.911448 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.895313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41595 (local address 127.0.0.1:48688)
0812 18:58:38.895843 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:38.895899 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:38.896858 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:38.897711 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:38.897716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:38.898074 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:38.898346 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:38.898359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:38.900222 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:38.900225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:38.901095 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:38.901106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:38.902121 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:38.902129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:38.902245 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:38.902808 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:38.902828 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:38.905905 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:38.908150 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:38.908162 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:38.908175 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:38.908421 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:38.908820 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:38.908823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:38.908825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:38.908947 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:38.909603 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:38.909610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:38.909766 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 18:58:38.909936 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 18:58:38.910398 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":144,"thread_start_us":154,"threads_started":1}
I20260812 18:58:38.913932 10694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:38.914865 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.914562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41402)
0812 18:58:38.914711 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:38.914788 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 18:58:38.915128 10635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
I20260812 18:58:38.916162 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.915894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41416)
0812 18:58:38.916007 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:38.916053 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 18:58:38.916350 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:43581: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
Entry for principal kudu/127.16.40.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab.
I20260812 18:58:38.924185 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:43581
--webserver_interface=127.16.40.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:58:38.965667 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } attempt: 1
I20260812 18:58:38.966640 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.966321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41424)
0812 18:58:38.966492 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:38.966556 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 18:58:38.967438 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:38.967216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41426)
0812 18:58:38.967321 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:38.967367 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 18:58:38.967626 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:43581: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
I20260812 18:58:39.035833 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } attempt: 2
I20260812 18:58:39.036828 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.036466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41440)
0812 18:58:39.036683 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.036748 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 18:58:39.037745 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.037493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41448)
0812 18:58:39.037626 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.037675 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 18:58:39.037992 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.253:43581: Network error: Client connection negotiation failed: client connection to 127.16.40.253:43581: connect: Connection refused (error 111)
W20260812 18:58:39.049552 10702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:39.049880 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:39.049940 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:39.050019 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:39.054451 10702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:39.054529 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:39.054550 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:39.054605 10702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:39.054636 10702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:39.060478 10702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.253.keytab
--principal=kudu/127.16.40.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.253:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.40.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:39.061816 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:39.062960 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:39.071830 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:39.071767 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:39.072255 10702 server_base.cc:1034] running on GCE node
W20260812 18:58:39.071779 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:39.072960 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:39.074227 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:39.075412 10702 hybrid_clock.cc:648] HybridClock initialized: now 1786561119075389 us; error 39 us; skew 500 ppm
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:39.078603 10702 init.cc:377] Logged in from keytab as kudu/127.16.40.253@KRBTEST.COM (short username kudu)
I20260812 18:58:39.079674 10702 webserver.cc:460] Webserver started at http://127.16.40.253:42419/ using document root <none> and password file <none>
I20260812 18:58:39.080323 10702 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:39.080447 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:39.080823 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:39.083451 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data/instance:
uuid: "3e5bf79d548247d0b187eb62383ce2ef"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "705e4579a71ddc01d9660d237863d044"
server_key_iv: "9bb5fde26db94aa68ee96c0b5311ceac"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.084201 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal/instance:
uuid: "3e5bf79d548247d0b187eb62383ce2ef"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "705e4579a71ddc01d9660d237863d044"
server_key_iv: "9bb5fde26db94aa68ee96c0b5311ceac"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.088368 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 18:58:39.091346 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:39.093065 10702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 18:58:39.093217 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal
uuid: "3e5bf79d548247d0b187eb62383ce2ef"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "705e4579a71ddc01d9660d237863d044"
server_key_iv: "9bb5fde26db94aa68ee96c0b5311ceac"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.093351 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:39.105871 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:39.109313 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:39.109535 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:39.116505 10702 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.253:43581
I20260812 18:58:39.116875 10782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.253:43581 every 8 connection(s)
I20260812 18:58:39.117615 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/data/info.pb
I20260812 18:58:39.120695 10784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:39.122985 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10702
I20260812 18:58:39.123098 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-1/wal/instance
I20260812 18:58:39.123420 16547 external_mini_cluster.cc:1428] Setting key 5a746f538d37f62bf34c27095249fa6e
I20260812 18:58:39.124961 10784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.129192 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } attempt: 3
WARNING: no policy specified for kudu/127.16.40.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.40.252@KRBTEST.COM" created.
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:39.143245 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.129689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41464)
0812 18:58:39.129883 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.129900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.129992 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.130577 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.130580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.130774 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.130905 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.130912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.132466 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.132473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.133061 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.133069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.134055 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.134061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.134336 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.134849 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.134862 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.137955 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.140818 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.140822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.140825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.141257 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.141700 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.141707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.141710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.141815 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.142768 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.142771 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.142844 (+ 73us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.142948 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.143104 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":18}
I20260812 18:58:39.143762 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.127007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.16.40.254:41595)
0812 18:58:39.127198 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.127204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.128249 (+ 1045us) server_negotiation.cc:407] Connection header received
0812 18:58:39.129386 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.129390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.129453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.129561 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.131348 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.132112 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.133725 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.134157 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.134270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.138887 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.138890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.138892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.138916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.141630 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.142198 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.142202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.142204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.142289 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.142581 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.142584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.142586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.142833 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.142943 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.143457 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.143599 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":69}
I20260812 18:58:39.145120 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.127389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41595 (local address 127.0.0.1:48692)
0812 18:58:39.127917 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.128079 (+ 162us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.129173 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.129745 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.129758 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.130347 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.131184 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.131206 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.132250 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.132257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.133578 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.133590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.134331 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.134336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.134444 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.135041 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.135066 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.138701 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.141780 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.141791 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.141798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.142085 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.142384 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.142388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.142390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.142477 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.142953 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.142962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.143162 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.143918 (+ 756us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.144163 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":70,"thread_start_us":147,"threads_started":1}
I20260812 18:58:39.145882 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.129793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.16.40.253:43581)
0812 18:58:39.130195 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.130204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.130223 (+ 19us) server_negotiation.cc:407] Connection header received
0812 18:58:39.130256 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.130260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.130412 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.130591 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.131055 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.132347 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.133227 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.133868 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.134069 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.138159 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.138171 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.138193 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.138229 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.140662 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.141392 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.141402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.141411 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.141582 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.141926 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.141933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.141937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.142641 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.142807 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.143692 (+ 885us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.144053 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":322,"thread_start_us":109,"threads_started":1}
I20260812 18:58:39.147735 10784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.148373 10694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.149281 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.148856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45190)
0812 18:58:39.149112 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.149180 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 18:58:39.150997 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.149848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45202)
0812 18:58:39.150824 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.150881 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":683,"threads_started":1}
W20260812 18:58:39.151257 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:39.159598 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.148366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41474)
0812 18:58:39.148512 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.148527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.148651 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.149823 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.149828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.150059 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.150202 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.150213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.151182 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.151188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.151779 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.151789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.152256 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.152259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.152389 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.152938 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.152950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.156165 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.157958 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.157963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.157965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.158221 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.158498 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.158503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.158504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.158609 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.159018 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.159022 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.159125 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.159235 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.159398 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":19}
I20260812 18:58:39.159607 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.148608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.16.40.253:43581)
0812 18:58:39.149402 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.149406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.149420 (+ 14us) server_negotiation.cc:407] Connection header received
0812 18:58:39.149459 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.149464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.149517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.149597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.150343 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.151066 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.151901 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.152149 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.152258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.156337 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.156342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.156344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.156373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.157839 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.158318 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.158325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.158330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.158407 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.158726 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.158730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.158731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.158916 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.159018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.159279 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.159474 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":691}
I20260812 18:58:39.160776 10784 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.161661 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.161390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45214)
0812 18:58:39.161518 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.161598 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 18:58:39.162238 10720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.163244 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.163025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45224)
0812 18:58:39.163136 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.163191 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
Entry for principal kudu/127.16.40.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab.
W20260812 18:58:39.163415 10784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.164443 16547 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
/tmp/dist-test-taskTWy6xo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:42957
--webserver_interface=127.16.40.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.40.212:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 18:58:39.182457 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 1
I20260812 18:58:39.183471 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.183135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45234)
0812 18:58:39.183319 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.183385 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 18:58:39.184408 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.184141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45250)
0812 18:58:39.184286 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.184340 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 18:58:39.184635 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.212738 10784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 1
I20260812 18:58:39.213811 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.213434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45260)
0812 18:58:39.213646 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.213723 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 18:58:39.214702 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.214434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45274)
0812 18:58:39.214557 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.214615 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 18:58:39.214849 10784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.259831 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 2
I20260812 18:58:39.260901 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.260531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45280)
0812 18:58:39.260740 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.260807 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 18:58:39.261754 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.261487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45294)
0812 18:58:39.261616 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.261679 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 18:58:39.261896 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.283025 10784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 2
I20260812 18:58:39.283967 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.283656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45296)
0812 18:58:39.283819 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.283885 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 18:58:39.284777 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.284524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45306)
0812 18:58:39.284659 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.284718 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 18:58:39.284902 10784 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
W20260812 18:58:39.290110 10796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 18:58:39.290474 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 18:58:39.290599 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 18:58:39.290704 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 18:58:39.295465 10796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 18:58:39.295578 10796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 18:58:39.295620 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 18:58:39.295678 10796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 18:58:39.295717 10796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 18:58:39.301877 10796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.40.212:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.40.254:41595,127.16.40.253:43581,127.16.40.252:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/krb5kdc/kudu_127.16.40.252.keytab
--principal=kudu/127.16.40.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.40.252:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.40.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 18:16:19 UTC on ccycloud.quasar-vfnacu.root.comops.site
build id 4583234
I20260812 18:58:39.303083 10796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 18:58:39.304159 10796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 18:58:39.313011 10802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:39.313030 10803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 18:58:39.313035 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 18:58:39.313305 10796 server_base.cc:1034] running on GCE node
I20260812 18:58:39.313977 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 18:58:39.315114 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 18:58:39.316305 10796 hybrid_clock.cc:648] HybridClock initialized: now 1786561119316293 us; error 29 us; skew 500 ppm
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 18:58:39.319175 10796 init.cc:377] Logged in from keytab as kudu/127.16.40.252@KRBTEST.COM (short username kudu)
I20260812 18:58:39.320065 10796 webserver.cc:460] Webserver started at http://127.16.40.252:46819/ using document root <none> and password file <none>
I20260812 18:58:39.320710 10796 fs_manager.cc:359] Metadata directory not provided
I20260812 18:58:39.320802 10796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 18:58:39.321043 10796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 18:58:39.323143 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data/instance:
uuid: "0b69fbb8867d475581d55f522d652982"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "8e1d86b603acb2da36b04ea47d6954ef"
server_key_iv: "980a4d366684c982423d48f8c67a7207"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.323720 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal/instance:
uuid: "0b69fbb8867d475581d55f522d652982"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "8e1d86b603acb2da36b04ea47d6954ef"
server_key_iv: "980a4d366684c982423d48f8c67a7207"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.327606 10796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 18:58:39.330145 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 18:58:39.331466 10796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 18:58:39.331624 10796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data,/tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal
uuid: "0b69fbb8867d475581d55f522d652982"
format_stamp: "Formatted at 2026-08-12 18:58:39 on dist-test-slave-rnrw"
server_key: "8e1d86b603acb2da36b04ea47d6954ef"
server_key_iv: "980a4d366684c982423d48f8c67a7207"
server_key_version: "encryptionkey@0"
I20260812 18:58:39.331745 10796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 18:58:39.358716 10796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 18:58:39.361063 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 3
I20260812 18:58:39.361917 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.361557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45322)
0812 18:58:39.361738 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.361802 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 18:58:39.362671 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.362444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45330)
0812 18:58:39.362552 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.362599 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 18:58:39.362797 10694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.40.252:42957: Network error: Client connection negotiation failed: client connection to 127.16.40.252:42957: connect: Connection refused (error 111)
I20260812 18:58:39.363026 10796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 18:58:39.363236 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 18:58:39.371215 10796 rpc_server.cc:307] RPC server started. Bound to: 127.16.40.252:42957
I20260812 18:58:39.371500 10872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.40.252:42957 every 8 connection(s)
I20260812 18:58:39.372347 10796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/data/info.pb
I20260812 18:58:39.373030 16547 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu as pid 10796
I20260812 18:58:39.373176 16547 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTWy6xo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786560811542710-16547-0/minicluster-data/master-2/wal/instance
I20260812 18:58:39.373493 16547 external_mini_cluster.cc:1428] Setting key a437ac9c298698f01c9a648e57437ec5
I20260812 18:58:39.375754 10873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 18:58:39.376081 10784 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 3
I20260812 18:58:39.380877 10873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.253@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:58:39.390120 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.376675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45348)
0812 18:58:39.376801 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.376817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.376902 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.381012 (+ 4110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.381018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.381220 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.381357 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.381365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.382520 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.382525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.383242 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.383260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.383938 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.383942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.384029 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.384643 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.384663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.386161 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.388239 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.388244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.388247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.388514 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.388843 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.388847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.388848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.388948 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.389566 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.389569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.389682 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.389787 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.389939 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:39.391503 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.374951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.16.40.252:42957)
0812 18:58:39.375404 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.375426 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.375474 (+ 48us) server_negotiation.cc:407] Connection header received
0812 18:58:39.376388 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.376413 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.376913 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.377279 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.378043 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.379550 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.380449 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.381170 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.381350 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.385048 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.385054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.385085 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.385120 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.387535 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.388338 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.388346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.388354 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.388585 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.388952 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.388957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.388960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.389465 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.389662 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.390099 (+ 437us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.390303 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"spinlock_wait_cycles":2048,"thread_start_us":120,"threads_started":1}
I20260812 18:58:39.391520 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.376642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.16.40.252:42957)
0812 18:58:39.380776 (+ 4134us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.380783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.380795 (+ 12us) server_negotiation.cc:407] Connection header received
0812 18:58:39.380825 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.380827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.380862 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.380970 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.381617 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.382372 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.383385 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.383715 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.383897 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.386330 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.386336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.386342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.386375 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.388128 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.388644 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.388647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.388649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.388734 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.389059 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.389062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.389064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.389452 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.389616 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.389957 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.390446 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":4069,"thread_start_us":105,"threads_started":1}
I20260812 18:58:39.394943 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.382948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.16.40.254:41595)
0812 18:58:39.383104 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.383109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.383787 (+ 678us) server_negotiation.cc:407] Connection header received
0812 18:58:39.383954 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.383958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.384009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.384062 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.385132 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.385871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.387364 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.387617 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.387741 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.390860 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.390864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.390867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.390901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.392818 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.393338 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.393341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.393342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.393417 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.393845 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.393849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.393850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.394113 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.394231 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.394624 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.394777 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 18:58:39.395558 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.382871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.254:41595 (local address 127.0.0.1:48704)
0812 18:58:39.383647 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.383707 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.383831 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.384269 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.384275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.384833 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.384989 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.384996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.385981 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.385984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.387038 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.387048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.387741 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.387745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.387864 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.388354 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.388365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.390680 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.392928 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.392936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.392941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.393235 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.393629 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.393633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.393634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.393733 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.394228 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.394235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.394366 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.395089 (+ 723us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.395321 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":155,"threads_started":1}
I20260812 18:58:39.396267 10784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: Bootstrap starting.
I20260812 18:58:39.397666 10873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.399744 10784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: Neither blocks nor log segments found. Creating new log.
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.254@KRBTEST.COM
I20260812 18:58:39.400861 10784 log.cc:824] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:39.403043 10784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: No bootstrap required, opened a new log
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:39.404883 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.394523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.16.40.254:41595)
0812 18:58:39.394845 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.394860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.394873 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:58:39.394905 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.394908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.394954 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.395837 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.395921 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.396470 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.397495 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.397772 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.397888 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.400836 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.400840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.400843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.400877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.402705 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.403239 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.403242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.403244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.403355 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.403716 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.403719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.403721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.403956 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.404182 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.404500 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.404682 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
I20260812 18:58:39.407562 10784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.407828 10784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5bf79d548247d0b187eb62383ce2ef, State: Initialized, Role: FOLLOWER
I20260812 18:58:39.408226 10784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.408511 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.398313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.253:43581 (local address 127.0.0.1:41488)
0812 18:58:39.398444 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.398457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.398537 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.398821 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.398827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.399022 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.399195 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.399205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.400173 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.400178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.401008 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.401022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.401621 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.401625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.401734 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.402903 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.402920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.404384 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.406712 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.406715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.406718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.407007 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.407329 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.407332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.407334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.407453 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.407978 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.407981 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.408077 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.408192 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.408344 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 18:58:39.408758 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.398377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.16.40.253:43581)
0812 18:58:39.398512 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.398516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.398531 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:58:39.398644 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.398648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.398694 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.398779 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.399334 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.400035 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.401120 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.401475 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.401600 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.404579 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.404583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.404586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.404625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.406596 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.407125 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.407128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.407130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.407221 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.407582 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.407585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.407587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.407839 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.407933 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.408248 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.408357 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 18:58:39.409638 10873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } has no permanent_uuid. Determining permanent_uuid...
I20260812 18:58:39.409819 10784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.40.253@KRBTEST.COM
I20260812 18:58:39.413172 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.413410 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: This master's current role is: FOLLOWER
W20260812 18:58:39.416500 10894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:39.416646 10894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:39.414040 10702 master_runner.cc:186] Error getting master registration for 127.16.40.254:41595: OK, instance_id { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" instance_seqno: 1786561118882378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:58:39.417436 10702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.252@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:58:39.419624 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.406197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.16.40.253:43581)
0812 18:58:39.406502 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.406511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.406524 (+ 13us) server_negotiation.cc:407] Connection header received
0812 18:58:39.406559 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.406562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.406603 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.406741 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.407541 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.408336 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.409393 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.409688 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.409805 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.413623 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.413628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.413631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.413660 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.416683 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.417898 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.417904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.417908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.418003 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.418346 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.418351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.418354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.418610 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.418826 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.419211 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.419371 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
I20260812 18:58:39.423084 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.410121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45350)
0812 18:58:39.410894 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.410912 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.411028 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.411354 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.411357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.411540 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.411654 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.411665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.412722 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.412726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.413442 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.413453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.415652 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.415658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.415747 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.417292 (+ 1545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.417308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.418743 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.421215 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.421219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.421221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.421473 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.421817 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.421820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.421822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.421909 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.422555 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.422558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.422654 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.422778 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.422932 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690}
I20260812 18:58:39.424190 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.410669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.16.40.252:42957)
0812 18:58:39.410797 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.410801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.410968 (+ 167us) server_negotiation.cc:407] Connection header received
0812 18:58:39.411124 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.411127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.411169 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.411222 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.411768 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.412590 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.414013 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.414342 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.414450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.418893 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.418897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.418900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.418922 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.421104 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.421580 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.421583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.421585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.421688 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.422161 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.422166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.422168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.422448 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.422561 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.422853 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.423240 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 18:58:39.424296 16547 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 18:58:39.425724 10664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6d9f4f7d0dc7404b961802b3a3808533"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48710
I20260812 18:58:39.427459 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: Bootstrap starting.
I20260812 18:58:39.430305 10873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:39.431281 10873 log.cc:824] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:39.433840 10873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: No bootstrap required, opened a new log
I20260812 18:58:39.438302 10873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.438611 10873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b69fbb8867d475581d55f522d652982, State: Initialized, Role: FOLLOWER
I20260812 18:58:39.438999 10873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.439582 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.439728 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.440474 10873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 18:58:39.446872 10912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:39.447072 10912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:39.444720 10796 master_runner.cc:186] Error getting master registration for 127.16.40.254:41595: OK, instance_id { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" instance_seqno: 1786561118882378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 18:58:39.448097 10796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 18:58:39.526983 10694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } attempt: 4
Aug 12 18:58:39 dist-test-slave-rnrw krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786561118, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.40.254@KRBTEST.COM for kudu/127.16.40.252@KRBTEST.COM
I20260812 18:58:39.538538 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.527737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.40.252:42957 (local address 127.0.0.1:45356)
0812 18:58:39.528089 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0812 18:58:39.528107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 18:58:39.528213 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 18:58:39.528876 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 18:58:39.528880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 18:58:39.529112 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 18:58:39.529248 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.529256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.530381 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.530385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.531224 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 18:58:39.531235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.531852 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.531855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 18:58:39.531946 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.532611 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 18:58:39.532628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 18:58:39.534391 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 18:58:39.536741 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.536745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.536747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.537019 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.537346 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 18:58:39.537349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 18:58:39.537351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 18:58:39.537449 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 18:58:39.537943 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 18:58:39.537947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 18:58:39.538060 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 18:58:39.538178 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 18:58:39.538336 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260812 18:58:39.539235 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.528390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.16.40.252:42957)
0812 18:58:39.528602 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.528609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.528630 (+ 21us) server_negotiation.cc:407] Connection header received
0812 18:58:39.528681 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.528684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.528737 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.528844 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.529394 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.530264 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.531363 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.531730 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.531848 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.534586 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.534591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.534594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.534623 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.536597 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.537138 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.537142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.537144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.537231 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.537553 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.537556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.537558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.537794 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.537895 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.538950 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.539083 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 18:58:39.543566 10694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: Bootstrap starting.
I20260812 18:58:39.546828 10694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: Neither blocks nor log segments found. Creating new log.
I20260812 18:58:39.547928 10694 log.cc:824] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: Log is configured to *not* fsync() on all Append() calls
I20260812 18:58:39.550671 10694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: No bootstrap required, opened a new log
I20260812 18:58:39.554255 10694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.554605 10694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9f4f7d0dc7404b961802b3a3808533, State: Initialized, Role: FOLLOWER
I20260812 18:58:39.554999 10694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.555872 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.556192 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.556813 10694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 18:58:39.556865 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 18:58:39.556977 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:39.558314 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.559307 10664 leader_election.cc:288] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [CANDIDATE]: Term 1 election: Requested vote from peers 3e5bf79d548247d0b187eb62383ce2ef (127.16.40.253:43581), 0b69fbb8867d475581d55f522d652982 (127.16.40.252:42957)
I20260812 18:58:39.561038 10750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d9f4f7d0dc7404b961802b3a3808533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e5bf79d548247d0b187eb62383ce2ef"
I20260812 18:58:39.561321 10750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:39.562723 10750 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9f4f7d0dc7404b961802b3a3808533 in term 1.
I20260812 18:58:39.563179 10843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d9f4f7d0dc7404b961802b3a3808533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b69fbb8867d475581d55f522d652982"
I20260812 18:58:39.563398 10843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [term 0 FOLLOWER]: Advancing to term 1
I20260812 18:58:39.564692 10843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9f4f7d0dc7404b961802b3a3808533 in term 1.
W20260812 18:58:39.563584 10928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 18:58:39.565415 10928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 18:58:39.566869 10634 leader_election.cc:302] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b69fbb8867d475581d55f522d652982, 6d9f4f7d0dc7404b961802b3a3808533; no voters:
I20260812 18:58:39.567672 10916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 18:58:39.567875 10916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [term 1 LEADER]: Becoming Leader. State: Replica: 6d9f4f7d0dc7404b961802b3a3808533, State: Running, Role: LEADER
I20260812 18:58:39.568229 10916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } }
I20260812 18:58:39.574875 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9f4f7d0dc7404b961802b3a3808533. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.575119 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:39.578123 10940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 18:58:39.581734 10940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 18:58:39.599660 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.566836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.16.40.254:41595)
0812 18:58:39.566970 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.566975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.576220 (+ 9245us) server_negotiation.cc:407] Connection header received
0812 18:58:39.576426 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.576430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.576488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.576560 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.577722 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.578517 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.592657 (+ 14140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.593045 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.593127 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.595173 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.595180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.595185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.595223 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.597136 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.597687 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.597692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.597695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.597795 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.598183 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.598188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.598193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.598465 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.598631 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.599294 (+ 663us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.599449 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 18:58:39.603788 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.577163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.16.40.253:43581)
0812 18:58:39.577313 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.577318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.578711 (+ 1393us) server_negotiation.cc:407] Connection header received
0812 18:58:39.578986 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.578990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.579046 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.579098 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.584394 (+ 5296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.585239 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.591110 (+ 5871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.591519 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.591615 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.596728 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.596736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.596740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.596774 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.598709 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.601482 (+ 2773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.601486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.601488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.601594 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.602269 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.602273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.602275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.602533 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.602657 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.603464 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.603606 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 18:58:39.609644 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:39.567482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.16.40.252:42957)
0812 18:58:39.567592 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.567596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.578479 (+ 10883us) server_negotiation.cc:407] Connection header received
0812 18:58:39.579231 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.579235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.579293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.579342 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.585402 (+ 6060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.586163 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.592442 (+ 6279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:39.592803 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:39.592885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:39.599156 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 18:58:39.599160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 18:58:39.599163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 18:58:39.599198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 18:58:39.601140 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.602978 (+ 1838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.602982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.602984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.603083 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 18:58:39.608290 (+ 5207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 18:58:39.608296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 18:58:39.608298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 18:58:39.608655 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 18:58:39.608840 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:39.609292 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 18:58:39.609452 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 18:58:39.610366 10750 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [term 1 FOLLOWER]: Refusing update from remote peer 6d9f4f7d0dc7404b961802b3a3808533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 18:58:39.611301 10725 tablet.cc:2369] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:58:39.612375 10843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [term 1 FOLLOWER]: Refusing update from remote peer 6d9f4f7d0dc7404b961802b3a3808533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 18:58:39.612313 10935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 18:58:39.614730 10821 tablet.cc:2369] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 18:58:39.616514 10935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 18:58:39.617050 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9f4f7d0dc7404b961802b3a3808533. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.617182 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.622184 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d9f4f7d0dc7404b961802b3a3808533. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.622336 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.622547 10935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.622720 10935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:39.624020 10882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.624161 10882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.624384 10943 mvcc.cc:205] Tried to move back new op lower bound from 7317754345908572160 to 7317754345756069888. Current Snapshot: MvccSnapshot[applied={T|T < 7317754345908572160}]
I20260812 18:58:39.624833 10944 mvcc.cc:205] Tried to move back new op lower bound from 7317754345908572160 to 7317754345756069888. Current Snapshot: MvccSnapshot[applied={T|T < 7317754345908572160}]
I20260812 18:58:39.625383 10940 catalog_manager.cc:1347] Generated new cluster ID: 61d2853d35fe4b84b0179ddabf506418
I20260812 18:58:39.625535 10940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 18:58:39.629249 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.629374 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533 [sys.catalog]: This master's current role is: LEADER
I20260812 18:58:39.630684 10897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d9f4f7d0dc7404b961802b3a3808533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f4f7d0dc7404b961802b3a3808533" member_type: VOTER last_known_addr { host: "127.16.40.254" port: 41595 } } peers { permanent_uuid: "3e5bf79d548247d0b187eb62383ce2ef" member_type: VOTER last_known_addr { host: "127.16.40.253" port: 43581 } } peers { permanent_uuid: "0b69fbb8867d475581d55f522d652982" member_type: VOTER last_known_addr { host: "127.16.40.252" port: 42957 } } }
I20260812 18:58:39.630801 10897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 18:58:39.636631 10946 mvcc.cc:205] Tried to move back new op lower bound from 7317754345908572160 to 7317754345756069888. Current Snapshot: MvccSnapshot[applied={T|T < 7317754345908572160 or (T in {7317754345908572160})}]
I20260812 18:58:39.649201 10940 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 18:58:39.650678 10940 catalog_manager.cc:1504] Loading token signing keys...
I20260812 18:58:39.664959 10940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d9f4f7d0dc7404b961802b3a3808533: Generated new TSK 0
I20260812 18:58:39.666469 10940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 18:58:39.693140 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.691820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.16.40.252:42957)
0812 18:58:39.691939 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.691943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.692018 (+ 75us) server_negotiation.cc:407] Connection header received
0812 18:58:39.692145 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.692148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.692201 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.692285 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.692974 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45368: BlockingRecv error: recv got EOF from 127.0.0.1:45368 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 18:58:39.693230 10879 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.691185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.16.40.254:41595)
0812 18:58:39.691331 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.691337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.691629 (+ 292us) server_negotiation.cc:407] Connection header received
0812 18:58:39.691894 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.691899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.691988 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.692055 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.692923 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48730: BlockingRecv error: recv got EOF from 127.0.0.1:48730 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 18:58:39.693746 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.691185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.16.40.253:43581)
0812 18:58:39.691331 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.691337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.692351 (+ 1014us) server_negotiation.cc:407] Connection header received
0812 18:58:39.692660 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.692665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.692730 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.692795 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.693621 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41514: BlockingRecv error: recv got EOF from 127.0.0.1:41514 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 18:58:39.697657 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.696207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.16.40.253:43581)
0812 18:58:39.696319 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.696322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.696511 (+ 189us) server_negotiation.cc:407] Connection header received
0812 18:58:39.696768 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.696771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.696840 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.696917 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.697519 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41520: BlockingRecv error: recv got EOF from 127.0.0.1:41520 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260812 18:58:39.702131 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.701051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.16.40.253:43581)
0812 18:58:39.701196 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.701202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.701224 (+ 22us) server_negotiation.cc:407] Connection header received
0812 18:58:39.701291 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.701294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.701354 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.701442 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.702012 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41530: BlockingRecv error: recv got EOF from 127.0.0.1:41530 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 18:58:39.709355 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.708086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.16.40.253:43581)
0812 18:58:39.708226 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.708231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.708347 (+ 116us) server_negotiation.cc:407] Connection header received
0812 18:58:39.708536 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.708602 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.708666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.708754 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.709227 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41540: BlockingRecv error: recv got EOF from 127.0.0.1:41540 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 18:58:39.716743 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.715442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.16.40.253:43581)
0812 18:58:39.715660 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.715666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.715762 (+ 96us) server_negotiation.cc:407] Connection header received
0812 18:58:39.715883 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.715886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.715949 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.716066 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.716610 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41552: BlockingRecv error: recv got EOF from 127.0.0.1:41552 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260812 18:58:39.728507 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.727026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.16.40.253:43581)
0812 18:58:39.727200 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.727205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.727417 (+ 212us) server_negotiation.cc:407] Connection header received
0812 18:58:39.727655 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.727658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.727718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.727817 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.728339 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41568: BlockingRecv error: recv got EOF from 127.0.0.1:41568 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 18:58:39.748162 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.746988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.16.40.253:43581)
0812 18:58:39.747139 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.747144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.747272 (+ 128us) server_negotiation.cc:407] Connection header received
0812 18:58:39.747414 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.747418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.747485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.747598 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.748032 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41574: BlockingRecv error: recv got EOF from 127.0.0.1:41574 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 18:58:39.783974 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.782740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.16.40.253:43581)
0812 18:58:39.782977 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.782980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.783059 (+ 79us) server_negotiation.cc:407] Connection header received
0812 18:58:39.783191 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.783194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.783248 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.783326 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.783813 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41576: BlockingRecv error: recv got EOF from 127.0.0.1:41576 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260812 18:58:39.851596 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.850349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.16.40.253:43581)
0812 18:58:39.850535 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.850539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.850665 (+ 126us) server_negotiation.cc:407] Connection header received
0812 18:58:39.850799 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.850802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.850852 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.850946 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.851483 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41584: BlockingRecv error: recv got EOF from 127.0.0.1:41584 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260812 18:58:39.983529 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:39.982077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.16.40.253:43581)
0812 18:58:39.982289 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:39.982295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:39.982473 (+ 178us) server_negotiation.cc:407] Connection header received
0812 18:58:39.982692 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:39.982695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:39.982750 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:39.982853 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:39.983401 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41592: BlockingRecv error: recv got EOF from 127.0.0.1:41592 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 18:58:40.243424 10881 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 18:58:40.242145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.16.40.253:43581)
0812 18:58:40.242318 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.242321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.242570 (+ 249us) server_negotiation.cc:407] Connection header received
0812 18:58:40.242705 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.242708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.242760 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.242836 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 18:58:40.243296 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41596: BlockingRecv error: recv got EOF from 127.0.0.1:41596 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 18:58:40.418073 10894 catalog_manager.cc:1259] Loaded cluster ID: 61d2853d35fe4b84b0179ddabf506418
I20260812 18:58:40.418171 10894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: loading cluster ID for follower catalog manager: success
I20260812 18:58:40.419914 10894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: acquiring CA information for follower catalog manager: success
I20260812 18:58:40.421170 10894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e5bf79d548247d0b187eb62383ce2ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:58:40.448676 10912 catalog_manager.cc:1259] Loaded cluster ID: 61d2853d35fe4b84b0179ddabf506418
I20260812 18:58:40.448758 10912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: loading cluster ID for follower catalog manager: success
I20260812 18:58:40.450507 10912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: acquiring CA information for follower catalog manager: success
I20260812 18:58:40.451712 10912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b69fbb8867d475581d55f522d652982: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 18:58:40.762362 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:40.758040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.16.40.253:43581)
0812 18:58:40.758291 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.758294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.758311 (+ 17us) server_negotiation.cc:407] Connection header received
0812 18:58:40.758419 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.758422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.758716 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.758843 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:40.759090 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.759771 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.760687 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.760909 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.761027 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:40.761427 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:40.761920 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:40.762056 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:40.762116 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 18:58:40.762182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260812 18:58:40.770012 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:40.766068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.16.40.252:42957)
0812 18:58:40.766321 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.766325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.766340 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:58:40.766435 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.766438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.766567 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.766678 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:40.766886 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.767521 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.768365 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.768718 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.768808 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:40.769145 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:40.769665 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:40.769775 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:40.769827 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 18:58:40.769879 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260812 18:58:40.778306 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:40.773204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.16.40.253:43581)
0812 18:58:40.773396 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.773401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.773801 (+ 400us) server_negotiation.cc:407] Connection header received
0812 18:58:40.773930 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.773934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.773991 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.774069 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:40.774440 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.775255 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.776799 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.777147 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.777237 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:40.777751 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:40.777833 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:40.777880 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:40.778064 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 18:58:40.778127 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 18:58:40.778306 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:40.773617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.16.40.252:42957)
0812 18:58:40.773735 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.773739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.773894 (+ 155us) server_negotiation.cc:407] Connection header received
0812 18:58:40.774076 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.774079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.774125 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.774173 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:40.774545 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.775280 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.776651 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.777086 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.777183 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:40.777714 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:40.777814 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:40.777866 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:40.778073 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 18:58:40.778143 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 18:58:40.778823 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 18:58:40.773149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.16.40.254:41595)
0812 18:58:40.773555 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0812 18:58:40.773559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 18:58:40.773574 (+ 15us) server_negotiation.cc:407] Connection header received
0812 18:58:40.773689 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 18:58:40.773692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 18:58:40.773877 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 18:58:40.774020 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 18:58:40.774309 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.775316 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.776799 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 18:58:40.777164 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 18:58:40.777254 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 18:58:40.777800 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 18:58:40.778176 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 18:58:40.778256 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 18:58:40.778593 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 18:58:40.778667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":67,"threads_started":1}
I20260812 18:58:40.783291 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10613
I20260812 18:58:40.795651 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10702
I20260812 18:58:40.810298 16547 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTWy6xo/build/debug/bin/kudu with pid 10796
2026-08-12T18:58:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3382 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6666 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (309239 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST